Jul 06 16:38:23.103780 ip-10-0-130-239 systemd[1]: Starting Kubernetes Kubelet... Jul 06 16:38:23.501319 ip-10-0-130-239 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:38:23.501319 ip-10-0-130-239 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 16:38:23.501319 ip-10-0-130-239 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:38:23.501319 ip-10-0-130-239 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 16:38:23.501319 ip-10-0-130-239 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:38:23.502395 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.502293 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 16:38:23.507047 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507024 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:23.507047 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507045 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:23.507047 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507049 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507053 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507057 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507060 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507065 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507069 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507074 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507078 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507081 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507084 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507087 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507090 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507093 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507096 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507098 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507101 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507104 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507107 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507110 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:23.507156 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507115 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507118 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507122 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507125 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507128 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507131 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507134 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507137 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507141 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507144 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507146 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507149 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507152 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507155 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507157 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507161 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507164 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507167 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507171 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507174 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:23.507666 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507177 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507179 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507182 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507185 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507187 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507190 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507193 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507195 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507198 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507200 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507203 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507206 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507208 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507211 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507215 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507218 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507221 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507223 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507226 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507229 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:23.508181 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507231 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507234 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507236 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507239 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507242 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507245 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507247 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507250 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507254 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507257 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507261 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507264 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507267 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507270 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507273 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507275 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507278 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507282 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507285 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507287 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:23.508712 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507290 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507293 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507295 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507298 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507301 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507304 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507307 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507310 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507313 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507316 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507319 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507323 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507327 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507330 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507333 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507336 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507339 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.507341 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508964 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:23.509243 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508972 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508976 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508979 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508983 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508987 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508990 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508994 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.508997 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509000 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509004 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509007 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509009 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509012 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509024 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509028 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509031 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509033 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509036 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509039 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509044 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509047 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:23.509764 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509050 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509053 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509056 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509059 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509062 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509064 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509067 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509070 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509073 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509075 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509078 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509081 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509084 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509086 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509089 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509092 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509102 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509105 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509108 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:23.510311 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509112 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509116 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509119 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509123 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509126 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509131 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509134 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509137 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509140 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509143 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509146 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509149 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509152 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509155 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509158 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509161 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509163 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509166 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509169 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:23.510847 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509172 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509175 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509177 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509180 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509183 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509188 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509192 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509194 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509197 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509200 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509202 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509206 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509209 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509211 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509215 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509218 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509221 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509223 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509226 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:23.511338 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509229 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509231 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509234 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509237 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509240 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509244 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509247 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509250 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509253 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509256 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509259 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509262 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509265 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509268 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509271 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509273 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509276 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509278 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509281 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509284 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:23.511858 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509745 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509751 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509754 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509757 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509761 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509765 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509768 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509771 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509774 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509778 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509781 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509784 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509787 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509790 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509792 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509795 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509798 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509802 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509805 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509809 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:23.512393 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509813 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509816 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509819 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509822 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509825 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509827 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509830 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509833 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509835 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509838 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509841 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509843 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509846 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509849 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509851 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509854 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509857 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509860 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509862 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509865 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:23.512967 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509868 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509871 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509873 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509876 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509878 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509881 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509884 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509886 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509889 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509893 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509895 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509898 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509901 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509903 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509906 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509909 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509914 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509918 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509921 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:23.513496 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509924 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509927 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509930 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509933 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509935 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509938 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509941 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509944 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509947 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509949 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509952 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509955 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509958 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509962 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509965 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509968 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509971 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509974 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509977 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:23.514006 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509979 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509985 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509988 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509992 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509995 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.509998 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510000 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510003 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510005 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510008 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510011 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510013 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510016 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510019 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510021 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510024 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510027 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510030 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510032 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510035 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:23.514546 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510038 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510112 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510120 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510127 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510134 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510143 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510148 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510155 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510162 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510165 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510169 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510173 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510176 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510181 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510184 2574 flags.go:64] FLAG: --cgroup-root="" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510188 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510191 2574 flags.go:64] FLAG: --client-ca-file="" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510194 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510197 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510203 2574 flags.go:64] FLAG: --cluster-domain="" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510207 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510210 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510213 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510217 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 16:38:23.515087 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510221 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510225 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510228 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510232 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510235 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510238 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510242 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510245 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510248 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510253 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510257 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510260 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510263 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510266 2574 flags.go:64] FLAG: --enable-server="true" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510270 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510275 2574 flags.go:64] FLAG: --event-burst="100" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510278 2574 flags.go:64] FLAG: --event-qps="50" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510281 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510284 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510291 2574 flags.go:64] FLAG: --eviction-hard="" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510295 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510299 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510302 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510306 2574 flags.go:64] FLAG: --eviction-soft="" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510309 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 16:38:23.515765 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510312 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510315 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510319 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510322 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510325 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510329 2574 flags.go:64] FLAG: --feature-gates="" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510334 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510338 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510341 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510345 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510348 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510352 2574 flags.go:64] FLAG: --help="false" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510355 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510358 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510361 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510364 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510368 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510371 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510375 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510377 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510380 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510383 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510387 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510391 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 16:38:23.516470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510394 2574 flags.go:64] FLAG: --kube-reserved="" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510397 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510401 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510405 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510409 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510412 2574 flags.go:64] FLAG: --lock-file="" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510415 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510418 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510421 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510433 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510436 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510439 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510442 2574 flags.go:64] FLAG: --logging-format="text" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510445 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510448 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510452 2574 flags.go:64] FLAG: --manifest-url="" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510454 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510459 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510462 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510466 2574 flags.go:64] FLAG: --max-pods="110" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510469 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510473 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510476 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510479 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510482 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 16:38:23.517134 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510485 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510489 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510497 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510500 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510503 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510507 2574 flags.go:64] FLAG: --pod-cidr="" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510511 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510519 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510522 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510525 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510542 2574 flags.go:64] FLAG: --port="10250" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510548 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510551 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-017b1b735f3254392" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510555 2574 flags.go:64] FLAG: --qos-reserved="" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510558 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510561 2574 flags.go:64] FLAG: --register-node="true" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510565 2574 flags.go:64] FLAG: --register-with-taints="" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510573 2574 flags.go:64] FLAG: --registry-burst="10" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510576 2574 flags.go:64] FLAG: --registry-qps="5" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510579 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510582 2574 flags.go:64] FLAG: --reserved-memory="" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510586 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510590 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510593 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510596 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 16:38:23.517859 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510599 2574 flags.go:64] FLAG: --runonce="false" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510602 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510605 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510608 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510611 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510614 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510617 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510620 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510623 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510626 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510629 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510632 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510636 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510639 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510642 2574 flags.go:64] FLAG: --system-cgroups="" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510645 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510651 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510656 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510660 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510665 2574 flags.go:64] FLAG: --tls-min-version="" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510668 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510671 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510674 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510677 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510680 2574 flags.go:64] FLAG: --v="2" Jul 06 16:38:23.518524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510685 2574 flags.go:64] FLAG: --version="false" Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510690 2574 flags.go:64] FLAG: --vmodule="" Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510694 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.510697 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510821 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510825 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510828 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510831 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510834 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510837 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510840 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510843 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510846 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510849 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510852 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510855 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510858 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510861 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510864 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510867 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:23.519205 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510870 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510873 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510876 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510879 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510883 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510887 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510889 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510892 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510896 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510900 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510903 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510908 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510911 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510915 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510918 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510921 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510924 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510927 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510930 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:23.520133 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510933 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510935 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510939 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510941 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510944 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510947 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510950 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510952 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510955 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510958 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510961 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510964 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510967 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510970 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510973 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510976 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510979 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510985 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510988 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:23.520845 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510991 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510994 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.510998 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511001 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511003 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511006 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511009 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511012 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511014 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511017 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511020 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511022 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511025 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511028 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511031 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511033 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511036 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511039 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511041 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511044 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:23.521482 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511047 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511049 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511052 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511055 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511058 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511061 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511063 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511066 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511069 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511072 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511077 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511080 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511083 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511086 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511088 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511091 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511094 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511096 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511100 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511103 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:23.522034 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511105 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511108 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511111 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511114 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.511116 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.511830 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.521665 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.521684 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.522367 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.522376 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522442 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522449 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522453 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522457 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522460 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:23.522591 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522463 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522466 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522468 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522471 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522475 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522478 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522481 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522484 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522487 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522490 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522493 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522497 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522500 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522503 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522506 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522508 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522511 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522514 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522517 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522520 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:23.523001 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522523 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522542 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522548 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522552 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522556 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522560 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522563 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522566 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522569 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522572 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522575 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522577 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522580 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522583 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522586 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522589 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522592 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522595 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522598 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:23.523519 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522601 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522604 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522607 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522610 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522613 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522616 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522619 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522621 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522624 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522627 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522629 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522633 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522638 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522641 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522643 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522646 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522649 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522652 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522655 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:23.524099 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522658 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522663 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522666 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522670 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522672 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522675 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522678 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522681 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522684 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522686 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522689 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522692 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522695 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522697 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522700 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522702 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522707 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522709 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522712 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:23.524642 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522715 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522717 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522720 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522723 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522726 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522728 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522731 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522734 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522737 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522739 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522742 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522745 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522748 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522751 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522754 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522757 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:23.525140 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522760 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.522765 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522916 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522922 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522926 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522929 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522932 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522935 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522939 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522941 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522946 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522951 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522955 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522958 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:23.525566 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522962 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522965 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522968 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522971 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522974 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522977 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522980 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522983 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522986 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522989 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522992 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522994 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.522997 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523000 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523004 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523007 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523010 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523013 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523016 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523019 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:23.525942 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523022 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523025 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523028 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523031 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523034 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523037 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523039 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523043 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523047 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523050 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523053 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523056 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523059 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523062 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523065 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523067 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523070 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523073 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523075 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:23.526475 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523078 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523081 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523084 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523086 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523089 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523093 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523096 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523099 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523102 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523105 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523108 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523111 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523113 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523116 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523119 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523122 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523125 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523129 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523132 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523136 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:23.526999 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523139 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523141 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523144 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523147 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523149 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523152 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523155 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523158 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523161 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523164 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523166 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523169 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523171 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523174 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523177 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523180 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523182 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523185 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523189 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523191 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:23.527523 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523194 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523197 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523200 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523203 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523206 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523209 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523212 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:23.523214 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.523219 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.523365 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 16:38:23.528149 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.527368 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 16:38:23.528878 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.528857 2574 server.go:1019] "Starting client certificate rotation" Jul 06 16:38:23.528996 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.528976 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 16:38:23.529027 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.529019 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 16:38:23.531494 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.531479 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 16:38:23.532787 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.532758 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 16:38:23.546900 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.546879 2574 log.go:25] "Validated CRI v1 runtime API" Jul 06 16:38:23.549271 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.549250 2574 log.go:25] "Validated CRI v1 image API" Jul 06 16:38:23.551714 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.551694 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 16:38:23.554232 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.554206 2574 fs.go:135] Filesystem UUIDs: map[279bdd9d-4887-4eb6-ad40-dcd344e8b8a4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c24508a0-9782-488e-b81c-1ffe79ee6ac4:/dev/nvme0n1p3] Jul 06 16:38:23.554346 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.554228 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 16:38:23.560466 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.560332 2574 manager.go:216] Machine: {Timestamp:2026-07-06 16:38:23.559162777 +0000 UTC m=+0.329054312 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100042 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e326aae4c3277b15f38cd5d1578a0 SystemUUID:ec2e326a-ae4c-3277-b15f-38cd5d1578a0 BootID:a0153f5b-d17f-44b8-98a7-52c946de06ba Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:e2:66:00:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:e2:66:00:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:59:8f:81:2d:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 16:38:23.560466 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.560449 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 16:38:23.560680 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.560606 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 16:38:23.561960 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.561916 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 16:38:23.562144 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.561968 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-239.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 16:38:23.562717 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.562705 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 16:38:23.562769 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.562720 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 16:38:23.562769 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.562728 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 16:38:23.564253 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.564242 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 16:38:23.565553 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.565542 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 16:38:23.568890 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.568771 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 06 16:38:23.569159 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.569139 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 16:38:23.570055 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.570031 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 06 16:38:23.570151 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.570079 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 16:38:23.570151 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.570104 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 16:38:23.570319 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.570298 2574 kubelet.go:398] "Adding apiserver pod source" Jul 06 16:38:23.570368 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.570349 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 16:38:23.570880 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.570862 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 16:38:23.570946 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.570920 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 16:38:23.573727 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.573704 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 16:38:23.575353 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.575332 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 16:38:23.575632 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.575618 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 16:38:23.575669 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.575652 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 16:38:23.576733 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576722 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 16:38:23.576774 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576739 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 16:38:23.576774 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576745 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 16:38:23.576774 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576752 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 16:38:23.576774 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576764 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 16:38:23.576774 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576770 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 16:38:23.576914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576778 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 16:38:23.576914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576783 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 16:38:23.576914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576791 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 16:38:23.576914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576797 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 16:38:23.576914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576807 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 16:38:23.576914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.576812 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 16:38:23.578099 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.578087 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 16:38:23.578135 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.578100 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 16:38:23.585939 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.585921 2574 server.go:1268] "Started kubelet" Jul 06 16:38:23.586065 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.586033 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 16:38:23.586239 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.586175 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 16:38:23.586365 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.586262 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 16:38:23.586896 ip-10-0-130-239 systemd[1]: Started Kubernetes Kubelet. Jul 06 16:38:23.587585 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.587544 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 16:38:23.588835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.588819 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 06 16:38:23.592259 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.592239 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-239.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 16:38:23.592398 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.592377 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 16:38:23.592506 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.592411 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 16:38:23.592603 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.592583 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 16:38:23.593700 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.593682 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 16:38:23.593740 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.593688 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 16:38:23.593851 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.593832 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:23.594621 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594600 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 16:38:23.594726 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594601 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 16:38:23.594726 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594646 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 16:38:23.594726 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594710 2574 factory.go:55] Registering systemd factory Jul 06 16:38:23.594726 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594726 2574 factory.go:223] Registration of the systemd container factory successfully Jul 06 16:38:23.594912 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594795 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 06 16:38:23.594912 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594804 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 06 16:38:23.594992 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594937 2574 factory.go:153] Registering CRI-O factory Jul 06 16:38:23.594992 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.594949 2574 factory.go:223] Registration of the crio container factory successfully Jul 06 16:38:23.595051 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.595003 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 16:38:23.595051 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.595025 2574 factory.go:103] Registering Raw factory Jul 06 16:38:23.595051 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.595037 2574 manager.go:1196] Started watching for new ooms in manager Jul 06 16:38:23.595589 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.595572 2574 manager.go:349] Starting recovery of all containers Jul 06 16:38:23.607910 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.607881 2574 manager.go:354] Recovery completed Jul 06 16:38:23.608357 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.608334 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p7g2q" Jul 06 16:38:23.609984 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.609957 2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 16:38:23.613296 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.613282 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:23.615035 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.615004 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 16:38:23.615133 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.615068 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 16:38:23.615392 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.615369 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:23.615447 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.615409 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:23.615447 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.615424 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:23.615965 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.615947 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 16:38:23.615965 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.615961 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:23.616066 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.615963 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 16:38:23.616066 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.615994 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 06 16:38:23.616137 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.615094 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-239.ec2.internal.18bfc0bee554eee9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-239.ec2.internal,UID:ip-10-0-130-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-239.ec2.internal,},FirstTimestamp:2026-07-06 16:38:23.585873641 +0000 UTC m=+0.355765177,LastTimestamp:2026-07-06 16:38:23.585873641 +0000 UTC m=+0.355765177,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-239.ec2.internal,}" Jul 06 16:38:23.616827 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.616809 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:23.616877 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.616841 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:23.616877 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.616851 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:23.617217 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.617205 2574 policy_none.go:49] "None policy: Start" Jul 06 16:38:23.617252 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.617223 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 16:38:23.617252 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.617234 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 16:38:23.618235 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.618152 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-239.ec2.internal.18bfc0bee7175588 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-239.ec2.internal,UID:ip-10-0-130-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-239.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-239.ec2.internal,},FirstTimestamp:2026-07-06 16:38:23.615391112 +0000 UTC m=+0.385282653,LastTimestamp:2026-07-06 16:38:23.615391112 +0000 UTC m=+0.385282653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-239.ec2.internal,}" Jul 06 16:38:23.618344 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.618333 2574 policy_none.go:47] "Start" Jul 06 16:38:23.625270 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.625252 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p7g2q" Jul 06 16:38:23.630505 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.630432 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-239.ec2.internal.18bfc0bee717b801 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-239.ec2.internal,UID:ip-10-0-130-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-239.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-239.ec2.internal,},FirstTimestamp:2026-07-06 16:38:23.615416321 +0000 UTC m=+0.385307860,LastTimestamp:2026-07-06 16:38:23.615416321 +0000 UTC m=+0.385307860,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-239.ec2.internal,}" Jul 06 16:38:23.654176 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.654159 2574 manager.go:337] "Starting Device Plugin manager" Jul 06 16:38:23.654281 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.654198 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 16:38:23.654281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.654210 2574 server.go:87] "Starting device plugin registration server" Jul 06 16:38:23.654519 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.654506 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 16:38:23.654595 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.654564 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 16:38:23.654703 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.654686 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 16:38:23.654788 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.654775 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 16:38:23.654788 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.654788 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 16:38:23.655373 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.655352 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 16:38:23.655506 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.655412 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:23.740241 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.740149 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 16:38:23.742523 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.741520 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 16:38:23.742523 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.741572 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 16:38:23.742523 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.741597 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 16:38:23.742523 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.741640 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 16:38:23.748780 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.745629 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:23.755665 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.755643 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:23.757710 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.757691 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:23.757788 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.757727 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:23.757788 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.757739 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:23.757788 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.757763 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.772266 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.772241 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.772354 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.772274 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-239.ec2.internal\": node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:23.820992 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.820962 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:23.842126 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.842084 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal"] Jul 06 16:38:23.842209 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.842180 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:23.843959 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.843934 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:23.844042 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.843979 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:23.844042 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.843995 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:23.846400 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.846385 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:23.846545 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.846519 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.846608 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.846554 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:23.847259 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.847242 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:23.847259 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.847254 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:23.847385 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.847270 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:23.847385 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.847280 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:23.847385 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.847280 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:23.847511 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.847432 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:23.850466 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.850440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.850580 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.850481 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:23.851336 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.851313 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:23.851401 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.851368 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:23.851401 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.851386 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:23.879547 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.879520 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-239.ec2.internal\" not found" node="ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.884281 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.884260 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-239.ec2.internal\" not found" node="ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.921912 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:23.921882 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:23.996403 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.996313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1089697870923d03089f4f6fa189bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal\" (UID: \"b1089697870923d03089f4f6fa189bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.996403 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.996352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1089697870923d03089f4f6fa189bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal\" (UID: \"b1089697870923d03089f4f6fa189bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:23.996403 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:23.996374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1115f28ae51a4317d8862e47dd2c12eb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-239.ec2.internal\" (UID: \"1115f28ae51a4317d8862e47dd2c12eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.022419 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:24.022389 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:24.097383 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.097351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1089697870923d03089f4f6fa189bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal\" (UID: \"b1089697870923d03089f4f6fa189bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.097383 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.097373 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1089697870923d03089f4f6fa189bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal\" (UID: \"b1089697870923d03089f4f6fa189bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.097604 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.097398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1089697870923d03089f4f6fa189bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal\" (UID: \"b1089697870923d03089f4f6fa189bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.097604 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.097432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1115f28ae51a4317d8862e47dd2c12eb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-239.ec2.internal\" (UID: \"1115f28ae51a4317d8862e47dd2c12eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.097604 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.097461 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1089697870923d03089f4f6fa189bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal\" (UID: \"b1089697870923d03089f4f6fa189bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.097604 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.097487 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1115f28ae51a4317d8862e47dd2c12eb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-239.ec2.internal\" (UID: \"1115f28ae51a4317d8862e47dd2c12eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.123825 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:24.123778 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:24.186772 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.186742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.186838 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.186750 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.224244 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:24.224194 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:24.324745 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:24.324644 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:24.426899 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:24.426841 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:24.448528 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.448496 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:24.466164 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.466138 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:24.527636 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:24.527582 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:24.528761 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.528743 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 16:38:24.528930 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.528905 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 16:38:24.528930 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.528920 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 16:38:24.529024 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.528950 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 16:38:24.628584 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:24.628492 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:24.628584 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.628480 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 16:33:24 +0000 UTC" deadline="2028-03-18 05:58:39.778895108 +0000 UTC" Jul 06 16:38:24.628584 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.628531 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14893h20m15.150368603s" Jul 06 16:38:24.730552 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:24.730511 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-239.ec2.internal\" not found" Jul 06 16:38:24.752962 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.752934 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:24.794211 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.794173 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.831749 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.831722 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 16:38:24.832954 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.832930 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" Jul 06 16:38:24.874365 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:24.874329 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 16:38:25.008022 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.007970 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1089697870923d03089f4f6fa189bd3.slice/crio-5a7afdbf5e968ad642fc240b46e1a8ef3ca7adbf85b28a45bb95e03e5a109448 WatchSource:0}: Error finding container 5a7afdbf5e968ad642fc240b46e1a8ef3ca7adbf85b28a45bb95e03e5a109448: Status 404 returned error can't find the container with id 5a7afdbf5e968ad642fc240b46e1a8ef3ca7adbf85b28a45bb95e03e5a109448 Jul 06 16:38:25.012704 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.012675 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:38:25.572299 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.572244 2574 apiserver.go:52] "Watching apiserver" Jul 06 16:38:25.580416 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.580388 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 16:38:25.583793 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.583765 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-8jpgx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal","openshift-multus/multus-additional-cni-plugins-sj9ll","openshift-network-operator/iptables-alerter-r44dw","openshift-ovn-kubernetes/ovnkube-node-pvsnk","kube-system/konnectivity-agent-5w5l8","kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp","openshift-cluster-node-tuning-operator/tuned-l9gc8","openshift-image-registry/node-ca-nflws","openshift-multus/multus-pf8rk","openshift-multus/network-metrics-daemon-lhbj8","openshift-network-diagnostics/network-check-target-hhtvk","kube-system/global-pull-secret-syncer-xrpmh"] Jul 06 16:38:25.587752 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.587727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.589985 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.589963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:25.590105 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.590058 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:25.592958 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.592938 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.594241 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.594024 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 16:38:25.595572 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.595368 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:25.595572 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.595462 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:25.598366 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.598327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.600767 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.600743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:25.603230 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.603212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.605678 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.605656 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.607038 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hwz\" (UniqueName: \"kubernetes.io/projected/c75e82f6-540f-4d90-be0a-ed9c9445152f-kube-api-access-s6hwz\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.607131 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-system-cni-dir\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.607131 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-os-release\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.607131 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7nz\" (UniqueName: \"kubernetes.io/projected/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-kube-api-access-xr7nz\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.607264 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c75e82f6-540f-4d90-be0a-ed9c9445152f-host\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.607264 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.607264 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c75e82f6-540f-4d90-be0a-ed9c9445152f-serviceca\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.607264 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.607425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.607425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjr5\" (UniqueName: \"kubernetes.io/projected/08928192-3dac-4b46-a00d-0339b9417aa7-kube-api-access-qsjr5\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:25.607425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:25.607425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cnibin\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.607425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.607425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.607383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:25.608277 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.608261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.610816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.610798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.613339 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.613320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.615301 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.615281 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 16:38:25.615401 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.615303 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 16:38:25.615471 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.615407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 16:38:25.615560 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.615542 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 16:38:25.615629 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.615613 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 16:38:25.615685 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.615653 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.615685 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.615680 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 16:38:25.615785 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.615731 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:25.616373 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.616357 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 16:38:25.616445 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.616393 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 16:38:25.616445 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.616394 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 16:38:25.616737 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.616721 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 16:38:25.616866 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.616852 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 16:38:25.616924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.616910 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 16:38:25.616961 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.616921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 16:38:25.617692 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.617664 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 16:38:25.617692 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.617681 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 16:38:25.617862 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.617777 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 16:38:25.617862 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.617825 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 16:38:25.617862 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.617829 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 16:38:25.618025 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.617935 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 16:38:25.618724 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.618691 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-td2wd\"" Jul 06 16:38:25.618724 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.618707 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 16:38:25.618724 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.618718 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 16:38:25.618937 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.618764 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 16:38:25.618937 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.618782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k2xmx\"" Jul 06 16:38:25.619031 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.618971 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 16:38:25.619031 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.618996 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 16:38:25.619031 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.619019 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mmvdt\"" Jul 06 16:38:25.619185 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.619038 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mvpjt\"" Jul 06 16:38:25.619185 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.618973 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lvln9\"" Jul 06 16:38:25.619185 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.619142 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w69mj\"" Jul 06 16:38:25.619336 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.619204 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 16:38:25.619336 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.619152 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 16:38:25.619336 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.619273 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 16:38:25.619576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.619395 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c6pvq\"" Jul 06 16:38:25.619806 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.619789 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-plr8h\"" Jul 06 16:38:25.623114 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.623094 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5rr97\"" Jul 06 16:38:25.650714 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.650689 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 16:38:25.684310 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.684265 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6p94h" Jul 06 16:38:25.696361 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.696328 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 16:38:25.707910 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.707881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c75e82f6-540f-4d90-be0a-ed9c9445152f-host\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.708078 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.707918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-log-socket\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.708078 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.707944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-conf-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.708078 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.707992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.708078 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708033 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c75e82f6-540f-4d90-be0a-ed9c9445152f-host\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.708291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-var-lib-kubelet\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.708291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95ef54ee-5605-4f93-9324-0794af019db5-etc-tuned\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.708291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-systemd-units\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.708291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-node-log\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.708291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-hostroot\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.708291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dcf852d-ac55-4a06-8597-f77c25cb65a5-agent-certs\") pod \"konnectivity-agent-5w5l8\" (UID: \"3dcf852d-ac55-4a06-8597-f77c25cb65a5\") " pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:25.708291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-socket-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708296 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79a6f67d-6571-4bbe-92e9-d640eb94ac19-dbus\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-slash\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-sys-fs\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-cnibin\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-sys\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c75e82f6-540f-4d90-be0a-ed9c9445152f-serviceca\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjr5\" (UniqueName: \"kubernetes.io/projected/08928192-3dac-4b46-a00d-0339b9417aa7-kube-api-access-qsjr5\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708497 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-kubelet\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-run-netns\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.708638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-etc-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-cni-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysctl-conf\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th79k\" (UniqueName: \"kubernetes.io/projected/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-kube-api-access-th79k\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708887 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c5044fe4-50b7-4819-9820-07b80fab8014-multus-daemon-config\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysconfig\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.708938 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.708937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysctl-d\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.709025 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs podName:08928192-3dac-4b46-a00d-0339b9417aa7 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:26.209002736 +0000 UTC m=+2.978894272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs") pod "network-metrics-daemon-lhbj8" (UID: "08928192-3dac-4b46-a00d-0339b9417aa7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709042 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c75e82f6-540f-4d90-be0a-ed9c9445152f-serviceca\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8rd\" (UniqueName: \"kubernetes.io/projected/04eeda10-18cb-4506-a057-e044524c5538-kube-api-access-4s8rd\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hwz\" (UniqueName: \"kubernetes.io/projected/c75e82f6-540f-4d90-be0a-ed9c9445152f-kube-api-access-s6hwz\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-cni-bin\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.709207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-cni-netd\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-multus-certs\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57xb\" (UniqueName: \"kubernetes.io/projected/c5044fe4-50b7-4819-9820-07b80fab8014-kube-api-access-s57xb\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskqw\" (UniqueName: \"kubernetes.io/projected/95ef54ee-5605-4f93-9324-0794af019db5-kube-api-access-cskqw\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-registration-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88tn\" (UniqueName: \"kubernetes.io/projected/536cf0ac-af55-4f89-a588-3c6ebb380f18-kube-api-access-j88tn\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-ovnkube-config\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-system-cni-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5044fe4-50b7-4819-9820-07b80fab8014-cni-binary-copy\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-k8s-cni-cncf-io\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-tmp-dir\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-systemd\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-os-release\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.710044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-etc-kubernetes\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cnibin\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-modprobe-d\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-systemd\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709863 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cnibin\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.709962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04eeda10-18cb-4506-a057-e044524c5538-iptables-alerter-script\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7nz\" (UniqueName: \"kubernetes.io/projected/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-kube-api-access-xr7nz\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04eeda10-18cb-4506-a057-e044524c5538-host-slash\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710125 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-hosts-file\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710154 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/466a9c09-be51-4bee-a5d2-67fb2284c831-ovn-node-metrics-cert\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710231 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-socket-dir-parent\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.710924 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-kubelet\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-env-overrides\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710314 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcq6\" (UniqueName: \"kubernetes.io/projected/466a9c09-be51-4bee-a5d2-67fb2284c831-kube-api-access-wxcq6\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-netns\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-cni-multus\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-run\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710409 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710477 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-lib-modules\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79a6f67d-6571-4bbe-92e9-d640eb94ac19-kubelet-config\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-cni-bin\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-kubernetes\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-host\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95ef54ee-5605-4f93-9324-0794af019db5-tmp\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-system-cni-dir\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.711725 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710748 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-system-cni-dir\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.712377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-os-release\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.712377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-var-lib-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.712377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-ovn\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.712377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710843 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-os-release\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.712377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-ovnkube-script-lib\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.712377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dcf852d-ac55-4a06-8597-f77c25cb65a5-konnectivity-ca\") pod \"konnectivity-agent-5w5l8\" (UID: \"3dcf852d-ac55-4a06-8597-f77c25cb65a5\") " pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:25.712377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.710909 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-device-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.725231 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.725197 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6p94h" Jul 06 16:38:25.736386 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.736341 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 16:38:25.738385 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.738356 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:25.740426 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.740395 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjr5\" (UniqueName: \"kubernetes.io/projected/08928192-3dac-4b46-a00d-0339b9417aa7-kube-api-access-qsjr5\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:25.745020 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.744981 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:25.745165 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.745030 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:25.745165 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.745046 2574 projected.go:196] Error preparing data for projected volume kube-api-access-lgf6k for pod openshift-network-diagnostics/network-check-target-hhtvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:25.745165 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.745132 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k podName:6b3ad089-857a-42f9-961f-42eb89b1a14b nodeName:}" failed. No retries permitted until 2026-07-06 16:38:26.245108142 +0000 UTC m=+3.014999663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lgf6k" (UniqueName: "kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k") pod "network-check-target-hhtvk" (UID: "6b3ad089-857a-42f9-961f-42eb89b1a14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:25.746724 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.746666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" event={"ID":"b1089697870923d03089f4f6fa189bd3","Type":"ContainerStarted","Data":"5a7afdbf5e968ad642fc240b46e1a8ef3ca7adbf85b28a45bb95e03e5a109448"} Jul 06 16:38:25.747220 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.747194 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hwz\" (UniqueName: \"kubernetes.io/projected/c75e82f6-540f-4d90-be0a-ed9c9445152f-kube-api-access-s6hwz\") pod \"node-ca-nflws\" (UID: \"c75e82f6-540f-4d90-be0a-ed9c9445152f\") " pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.747312 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.747230 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7nz\" (UniqueName: \"kubernetes.io/projected/d48bd494-815a-4f2c-bd82-72cbbe63ca5c-kube-api-access-xr7nz\") pod \"multus-additional-cni-plugins-sj9ll\" (UID: \"d48bd494-815a-4f2c-bd82-72cbbe63ca5c\") " pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.811501 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dcf852d-ac55-4a06-8597-f77c25cb65a5-agent-certs\") pod \"konnectivity-agent-5w5l8\" (UID: \"3dcf852d-ac55-4a06-8597-f77c25cb65a5\") " pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:25.811708 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-socket-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.811708 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811554 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79a6f67d-6571-4bbe-92e9-d640eb94ac19-dbus\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.811708 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-slash\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.811708 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.811708 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-sys-fs\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.811708 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-cnibin\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.811708 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-sys\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.811708 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-kubelet\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-run-netns\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-etc-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-cni-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysctl-conf\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th79k\" (UniqueName: \"kubernetes.io/projected/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-kube-api-access-th79k\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811930 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-cnibin\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c5044fe4-50b7-4819-9820-07b80fab8014-multus-daemon-config\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.811974 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-slash\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysconfig\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812018 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-sys-fs\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812031 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-sys\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-kubelet\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812062 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-etc-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812098 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.812106 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812115 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-socket-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812125 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-cni-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812159 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysconfig\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812163 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysctl-d\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812176 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79a6f67d-6571-4bbe-92e9-d640eb94ac19-dbus\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812190 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysctl-d\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812201 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-run-netns\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8rd\" (UniqueName: \"kubernetes.io/projected/04eeda10-18cb-4506-a057-e044524c5538-kube-api-access-4s8rd\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-cni-bin\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-cni-netd\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-multus-certs\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57xb\" (UniqueName: \"kubernetes.io/projected/c5044fe4-50b7-4819-9820-07b80fab8014-kube-api-access-s57xb\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812327 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-sysctl-conf\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskqw\" (UniqueName: \"kubernetes.io/projected/95ef54ee-5605-4f93-9324-0794af019db5-kube-api-access-cskqw\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812357 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-cni-netd\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-registration-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.813037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812375 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-multus-certs\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812406 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-cni-bin\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88tn\" (UniqueName: \"kubernetes.io/projected/536cf0ac-af55-4f89-a588-3c6ebb380f18-kube-api-access-j88tn\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812462 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-registration-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-ovnkube-config\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812609 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c5044fe4-50b7-4819-9820-07b80fab8014-multus-daemon-config\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-system-cni-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5044fe4-50b7-4819-9820-07b80fab8014-cni-binary-copy\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-k8s-cni-cncf-io\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-tmp-dir\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-systemd\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812828 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-systemd\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812834 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-k8s-cni-cncf-io\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-os-release\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-etc-kubernetes\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-modprobe-d\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812947 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-os-release\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812943 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-systemd\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.813921 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812999 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-etc-kubernetes\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813016 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-systemd\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813032 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-ovnkube-config\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.812664 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-system-cni-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04eeda10-18cb-4506-a057-e044524c5538-iptables-alerter-script\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04eeda10-18cb-4506-a057-e044524c5538-host-slash\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813123 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-tmp-dir\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-hosts-file\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813191 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/466a9c09-be51-4bee-a5d2-67fb2284c831-ovn-node-metrics-cert\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813124 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-modprobe-d\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813226 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04eeda10-18cb-4506-a057-e044524c5538-host-slash\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-socket-dir-parent\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813256 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-kubelet\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813265 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-hosts-file\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.814883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-env-overrides\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813277 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-socket-dir-parent\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcq6\" (UniqueName: \"kubernetes.io/projected/466a9c09-be51-4bee-a5d2-67fb2284c831-kube-api-access-wxcq6\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-netns\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-cni-multus\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-run\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-lib-modules\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79a6f67d-6571-4bbe-92e9-d640eb94ac19-kubelet-config\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-cni-bin\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813516 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-kubernetes\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-host\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95ef54ee-5605-4f93-9324-0794af019db5-tmp\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813625 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-env-overrides\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813629 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04eeda10-18cb-4506-a057-e044524c5538-iptables-alerter-script\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-var-lib-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-ovn\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813698 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-kubelet\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.815835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813703 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79a6f67d-6571-4bbe-92e9-d640eb94ac19-kubelet-config\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813717 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-run-ovn\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813718 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-cni-multus\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813750 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-run\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813761 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-var-lib-cni-bin\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-ovnkube-script-lib\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813789 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5044fe4-50b7-4819-9820-07b80fab8014-cni-binary-copy\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dcf852d-ac55-4a06-8597-f77c25cb65a5-konnectivity-ca\") pod \"konnectivity-agent-5w5l8\" (UID: \"3dcf852d-ac55-4a06-8597-f77c25cb65a5\") " pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813821 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-lib-modules\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813846 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-host-run-netns\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-device-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813896 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-var-lib-openvswitch\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-log-socket\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813934 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-log-socket\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-conf-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.813205 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.813998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-var-lib-kubelet\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.816421 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814017 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:25.814028 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret podName:79a6f67d-6571-4bbe-92e9-d640eb94ac19 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:26.314009485 +0000 UTC m=+3.083901012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret") pod "global-pull-secret-syncer-xrpmh" (UID: "79a6f67d-6571-4bbe-92e9-d640eb94ac19") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814048 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-var-lib-kubelet\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95ef54ee-5605-4f93-9324-0794af019db5-etc-tuned\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-systemd-units\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-node-log\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814128 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-hostroot\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814188 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/536cf0ac-af55-4f89-a588-3c6ebb380f18-device-dir\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814211 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-etc-kubernetes\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814238 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95ef54ee-5605-4f93-9324-0794af019db5-host\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814241 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-hostroot\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814255 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-systemd-units\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814090 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5044fe4-50b7-4819-9820-07b80fab8014-multus-conf-dir\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814276 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/466a9c09-be51-4bee-a5d2-67fb2284c831-node-log\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.814265 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/466a9c09-be51-4bee-a5d2-67fb2284c831-ovnkube-script-lib\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.815040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dcf852d-ac55-4a06-8597-f77c25cb65a5-konnectivity-ca\") pod \"konnectivity-agent-5w5l8\" (UID: \"3dcf852d-ac55-4a06-8597-f77c25cb65a5\") " pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:25.817451 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.815445 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dcf852d-ac55-4a06-8597-f77c25cb65a5-agent-certs\") pod \"konnectivity-agent-5w5l8\" (UID: \"3dcf852d-ac55-4a06-8597-f77c25cb65a5\") " pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:25.818329 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.816552 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95ef54ee-5605-4f93-9324-0794af019db5-tmp\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.818329 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.816740 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95ef54ee-5605-4f93-9324-0794af019db5-etc-tuned\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.818329 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.816795 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/466a9c09-be51-4bee-a5d2-67fb2284c831-ovn-node-metrics-cert\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.833494 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.833403 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcq6\" (UniqueName: \"kubernetes.io/projected/466a9c09-be51-4bee-a5d2-67fb2284c831-kube-api-access-wxcq6\") pod \"ovnkube-node-pvsnk\" (UID: \"466a9c09-be51-4bee-a5d2-67fb2284c831\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.833494 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.833402 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57xb\" (UniqueName: \"kubernetes.io/projected/c5044fe4-50b7-4819-9820-07b80fab8014-kube-api-access-s57xb\") pod \"multus-pf8rk\" (UID: \"c5044fe4-50b7-4819-9820-07b80fab8014\") " pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.833786 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.833762 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88tn\" (UniqueName: \"kubernetes.io/projected/536cf0ac-af55-4f89-a588-3c6ebb380f18-kube-api-access-j88tn\") pod \"aws-ebs-csi-driver-node-7mccp\" (UID: \"536cf0ac-af55-4f89-a588-3c6ebb380f18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.833849 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.833791 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th79k\" (UniqueName: \"kubernetes.io/projected/9b13ab05-cd28-4ff2-9b5f-92d2c511bff5-kube-api-access-th79k\") pod \"node-resolver-8jpgx\" (UID: \"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5\") " pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.841387 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.841356 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8rd\" (UniqueName: \"kubernetes.io/projected/04eeda10-18cb-4506-a057-e044524c5538-kube-api-access-4s8rd\") pod \"iptables-alerter-r44dw\" (UID: \"04eeda10-18cb-4506-a057-e044524c5538\") " pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.843559 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.843535 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskqw\" (UniqueName: \"kubernetes.io/projected/95ef54ee-5605-4f93-9324-0794af019db5-kube-api-access-cskqw\") pod \"tuned-l9gc8\" (UID: \"95ef54ee-5605-4f93-9324-0794af019db5\") " pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.897720 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.897684 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nflws" Jul 06 16:38:25.905524 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.905500 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" Jul 06 16:38:25.909832 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.909797 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75e82f6_540f_4d90_be0a_ed9c9445152f.slice/crio-bebe56504523115c0659ee8f13fc09a143e793c666aae1dc874dd5f1e9555d82 WatchSource:0}: Error finding container bebe56504523115c0659ee8f13fc09a143e793c666aae1dc874dd5f1e9555d82: Status 404 returned error can't find the container with id bebe56504523115c0659ee8f13fc09a143e793c666aae1dc874dd5f1e9555d82 Jul 06 16:38:25.912554 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.912498 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:25.917988 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.917943 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:25.925563 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.925538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pf8rk" Jul 06 16:38:25.927203 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.927171 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466a9c09_be51_4bee_a5d2_67fb2284c831.slice/crio-0a3f9533076b6f47ab8766f8d0dd638eb2e243abc33bc2ad54b4d19eed09c4a7 WatchSource:0}: Error finding container 0a3f9533076b6f47ab8766f8d0dd638eb2e243abc33bc2ad54b4d19eed09c4a7: Status 404 returned error can't find the container with id 0a3f9533076b6f47ab8766f8d0dd638eb2e243abc33bc2ad54b4d19eed09c4a7 Jul 06 16:38:25.929989 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.929697 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48bd494_815a_4f2c_bd82_72cbbe63ca5c.slice/crio-3ad18a52f94ce7f43a0c84d2f6d968181e51cb0579fb368322b7d2ff17928d5f WatchSource:0}: Error finding container 3ad18a52f94ce7f43a0c84d2f6d968181e51cb0579fb368322b7d2ff17928d5f: Status 404 returned error can't find the container with id 3ad18a52f94ce7f43a0c84d2f6d968181e51cb0579fb368322b7d2ff17928d5f Jul 06 16:38:25.931991 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.931963 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcf852d_ac55_4a06_8597_f77c25cb65a5.slice/crio-ac848aabfd654f44d3fd9321fd4cf3170ad37f4f749a9cdb047ca71c7d1382e4 WatchSource:0}: Error finding container ac848aabfd654f44d3fd9321fd4cf3170ad37f4f749a9cdb047ca71c7d1382e4: Status 404 returned error can't find the container with id ac848aabfd654f44d3fd9321fd4cf3170ad37f4f749a9cdb047ca71c7d1382e4 Jul 06 16:38:25.932752 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.932549 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" Jul 06 16:38:25.938903 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.938875 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5044fe4_50b7_4819_9820_07b80fab8014.slice/crio-39e623cd8596f12e1fbd526f55ca78cc4a689d8ff706fcdf7e4765ec17ade03a WatchSource:0}: Error finding container 39e623cd8596f12e1fbd526f55ca78cc4a689d8ff706fcdf7e4765ec17ade03a: Status 404 returned error can't find the container with id 39e623cd8596f12e1fbd526f55ca78cc4a689d8ff706fcdf7e4765ec17ade03a Jul 06 16:38:25.942348 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.942328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" Jul 06 16:38:25.946426 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.946394 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536cf0ac_af55_4f89_a588_3c6ebb380f18.slice/crio-7c02e182a46809a5ef01cfd63d4a3ed4d558eec5fb55ba39d1432be1e107fc45 WatchSource:0}: Error finding container 7c02e182a46809a5ef01cfd63d4a3ed4d558eec5fb55ba39d1432be1e107fc45: Status 404 returned error can't find the container with id 7c02e182a46809a5ef01cfd63d4a3ed4d558eec5fb55ba39d1432be1e107fc45 Jul 06 16:38:25.952460 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.952436 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r44dw" Jul 06 16:38:25.953407 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.953378 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ef54ee_5605_4f93_9324_0794af019db5.slice/crio-527703b10c58ab5ea43b12118667f3d0b54ad537bb53bf99088312102494318b WatchSource:0}: Error finding container 527703b10c58ab5ea43b12118667f3d0b54ad537bb53bf99088312102494318b: Status 404 returned error can't find the container with id 527703b10c58ab5ea43b12118667f3d0b54ad537bb53bf99088312102494318b Jul 06 16:38:25.957603 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:25.957582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jpgx" Jul 06 16:38:25.963367 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.963344 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04eeda10_18cb_4506_a057_e044524c5538.slice/crio-6505a22b32195626a76f5f97c4d26f51cdddc1e362bf57c264dc852d1ab29f2c WatchSource:0}: Error finding container 6505a22b32195626a76f5f97c4d26f51cdddc1e362bf57c264dc852d1ab29f2c: Status 404 returned error can't find the container with id 6505a22b32195626a76f5f97c4d26f51cdddc1e362bf57c264dc852d1ab29f2c Jul 06 16:38:25.967520 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:25.967491 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b13ab05_cd28_4ff2_9b5f_92d2c511bff5.slice/crio-2c56bd5123e1899e2abd1138717bed1be29380872db3936f55ffa373a7f005e6 WatchSource:0}: Error finding container 2c56bd5123e1899e2abd1138717bed1be29380872db3936f55ffa373a7f005e6: Status 404 returned error can't find the container with id 2c56bd5123e1899e2abd1138717bed1be29380872db3936f55ffa373a7f005e6 Jul 06 16:38:26.216664 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.216564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:26.216827 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:26.216682 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:26.216827 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:26.216739 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs podName:08928192-3dac-4b46-a00d-0339b9417aa7 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:27.216725792 +0000 UTC m=+3.986617313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs") pod "network-metrics-daemon-lhbj8" (UID: "08928192-3dac-4b46-a00d-0339b9417aa7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:26.317894 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.317841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:26.318056 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.317929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:26.318056 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:26.318006 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:26.318056 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:26.318040 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:26.318056 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:26.318055 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:26.318268 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:26.318082 2574 projected.go:196] Error preparing data for projected volume kube-api-access-lgf6k for pod openshift-network-diagnostics/network-check-target-hhtvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:26.318268 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:26.318084 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret podName:79a6f67d-6571-4bbe-92e9-d640eb94ac19 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:27.318061446 +0000 UTC m=+4.087952972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret") pod "global-pull-secret-syncer-xrpmh" (UID: "79a6f67d-6571-4bbe-92e9-d640eb94ac19") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:26.318268 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:26.318129 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k podName:6b3ad089-857a-42f9-961f-42eb89b1a14b nodeName:}" failed. No retries permitted until 2026-07-06 16:38:27.31811762 +0000 UTC m=+4.088009141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgf6k" (UniqueName: "kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k") pod "network-check-target-hhtvk" (UID: "6b3ad089-857a-42f9-961f-42eb89b1a14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:26.730086 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.730044 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 16:33:26 +0000 UTC" deadline="2027-12-05 15:42:31.134799362 +0000 UTC" Jul 06 16:38:26.730086 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.730081 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12407h4m4.404722405s" Jul 06 16:38:26.750782 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.750737 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nflws" event={"ID":"c75e82f6-540f-4d90-be0a-ed9c9445152f","Type":"ContainerStarted","Data":"bebe56504523115c0659ee8f13fc09a143e793c666aae1dc874dd5f1e9555d82"} Jul 06 16:38:26.752146 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.752082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r44dw" event={"ID":"04eeda10-18cb-4506-a057-e044524c5538","Type":"ContainerStarted","Data":"6505a22b32195626a76f5f97c4d26f51cdddc1e362bf57c264dc852d1ab29f2c"} Jul 06 16:38:26.753680 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.753608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" event={"ID":"536cf0ac-af55-4f89-a588-3c6ebb380f18","Type":"ContainerStarted","Data":"7c02e182a46809a5ef01cfd63d4a3ed4d558eec5fb55ba39d1432be1e107fc45"} Jul 06 16:38:26.755132 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.755093 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pf8rk" event={"ID":"c5044fe4-50b7-4819-9820-07b80fab8014","Type":"ContainerStarted","Data":"39e623cd8596f12e1fbd526f55ca78cc4a689d8ff706fcdf7e4765ec17ade03a"} Jul 06 16:38:26.756751 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.756697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5w5l8" event={"ID":"3dcf852d-ac55-4a06-8597-f77c25cb65a5","Type":"ContainerStarted","Data":"ac848aabfd654f44d3fd9321fd4cf3170ad37f4f749a9cdb047ca71c7d1382e4"} Jul 06 16:38:26.758056 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.758026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" event={"ID":"d48bd494-815a-4f2c-bd82-72cbbe63ca5c","Type":"ContainerStarted","Data":"3ad18a52f94ce7f43a0c84d2f6d968181e51cb0579fb368322b7d2ff17928d5f"} Jul 06 16:38:26.759319 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.759273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jpgx" event={"ID":"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5","Type":"ContainerStarted","Data":"2c56bd5123e1899e2abd1138717bed1be29380872db3936f55ffa373a7f005e6"} Jul 06 16:38:26.760664 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.760639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" event={"ID":"95ef54ee-5605-4f93-9324-0794af019db5","Type":"ContainerStarted","Data":"527703b10c58ab5ea43b12118667f3d0b54ad537bb53bf99088312102494318b"} Jul 06 16:38:26.761981 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.761955 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"0a3f9533076b6f47ab8766f8d0dd638eb2e243abc33bc2ad54b4d19eed09c4a7"} Jul 06 16:38:26.832730 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:38:26.832630 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1115f28ae51a4317d8862e47dd2c12eb.slice/crio-6d7b89cb4479af22c39c51186d807872ac64ceab5ecbb606fb24de185d23ca7c WatchSource:0}: Error finding container 6d7b89cb4479af22c39c51186d807872ac64ceab5ecbb606fb24de185d23ca7c: Status 404 returned error can't find the container with id 6d7b89cb4479af22c39c51186d807872ac64ceab5ecbb606fb24de185d23ca7c Jul 06 16:38:26.929372 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:26.929312 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:27.122179 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.122145 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:27.226138 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.225869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:27.226373 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.226159 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:27.226373 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.226233 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs podName:08928192-3dac-4b46-a00d-0339b9417aa7 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:29.226212793 +0000 UTC m=+5.996104328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs") pod "network-metrics-daemon-lhbj8" (UID: "08928192-3dac-4b46-a00d-0339b9417aa7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:27.326928 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.326533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:27.326928 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.326630 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:27.326928 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.326745 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:27.326928 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.326796 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:27.326928 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.326815 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:27.326928 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.326830 2574 projected.go:196] Error preparing data for projected volume kube-api-access-lgf6k for pod openshift-network-diagnostics/network-check-target-hhtvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:27.326928 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.326830 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret podName:79a6f67d-6571-4bbe-92e9-d640eb94ac19 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:29.326807607 +0000 UTC m=+6.096699127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret") pod "global-pull-secret-syncer-xrpmh" (UID: "79a6f67d-6571-4bbe-92e9-d640eb94ac19") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:27.326928 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.326898 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k podName:6b3ad089-857a-42f9-961f-42eb89b1a14b nodeName:}" failed. No retries permitted until 2026-07-06 16:38:29.326880452 +0000 UTC m=+6.096771978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgf6k" (UniqueName: "kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k") pod "network-check-target-hhtvk" (UID: "6b3ad089-857a-42f9-961f-42eb89b1a14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:27.730645 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.730560 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 16:33:26 +0000 UTC" deadline="2028-03-10 09:43:10.031020458 +0000 UTC" Jul 06 16:38:27.730645 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.730641 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14705h4m42.300383852s" Jul 06 16:38:27.744806 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.744711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:27.744977 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.744855 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:27.746038 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.745269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:27.746038 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.745366 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:27.746038 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.745580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:27.746038 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:27.745681 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:27.781489 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.780643 2574 generic.go:356] "Generic (PLEG): container finished" podID="b1089697870923d03089f4f6fa189bd3" containerID="225d35faba5e26a26411e1847c31eac7840c35d8c206b2a4fdff214d623b8b0d" exitCode=0 Jul 06 16:38:27.781489 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.780761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" event={"ID":"b1089697870923d03089f4f6fa189bd3","Type":"ContainerDied","Data":"225d35faba5e26a26411e1847c31eac7840c35d8c206b2a4fdff214d623b8b0d"} Jul 06 16:38:27.791884 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:27.791842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" event={"ID":"1115f28ae51a4317d8862e47dd2c12eb","Type":"ContainerStarted","Data":"6d7b89cb4479af22c39c51186d807872ac64ceab5ecbb606fb24de185d23ca7c"} Jul 06 16:38:29.245245 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:29.245152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:29.245805 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.245389 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:29.245805 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.245475 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs podName:08928192-3dac-4b46-a00d-0339b9417aa7 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:33.245455497 +0000 UTC m=+10.015347023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs") pod "network-metrics-daemon-lhbj8" (UID: "08928192-3dac-4b46-a00d-0339b9417aa7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:29.346903 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:29.346215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:29.346903 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:29.346301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:29.346903 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.346456 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:29.346903 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.346540 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret podName:79a6f67d-6571-4bbe-92e9-d640eb94ac19 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:33.346521276 +0000 UTC m=+10.116412798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret") pod "global-pull-secret-syncer-xrpmh" (UID: "79a6f67d-6571-4bbe-92e9-d640eb94ac19") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:29.346903 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.346734 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:29.346903 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.346760 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:29.346903 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.346775 2574 projected.go:196] Error preparing data for projected volume kube-api-access-lgf6k for pod openshift-network-diagnostics/network-check-target-hhtvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:29.346903 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.346855 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k podName:6b3ad089-857a-42f9-961f-42eb89b1a14b nodeName:}" failed. No retries permitted until 2026-07-06 16:38:33.34683824 +0000 UTC m=+10.116729780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgf6k" (UniqueName: "kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k") pod "network-check-target-hhtvk" (UID: "6b3ad089-857a-42f9-961f-42eb89b1a14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:29.742729 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:29.742624 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:29.742909 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:29.742624 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:29.742909 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.742776 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:29.742909 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.742889 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:29.743096 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:29.742624 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:29.743149 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:29.743041 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:30.802431 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:30.802387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" event={"ID":"b1089697870923d03089f4f6fa189bd3","Type":"ContainerStarted","Data":"d9f925fc2c8addeff515a6dbcd049d5c85081296b1c22fd65ed1d60ab2635b3d"} Jul 06 16:38:31.742663 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:31.742576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:31.742842 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:31.742576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:31.742842 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:31.742726 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:31.742959 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:31.742859 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:31.742959 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:31.742576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:31.743065 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:31.742965 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:33.282445 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:33.282401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:33.282964 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.282600 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:33.282964 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.282685 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs podName:08928192-3dac-4b46-a00d-0339b9417aa7 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:41.282663423 +0000 UTC m=+18.052554952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs") pod "network-metrics-daemon-lhbj8" (UID: "08928192-3dac-4b46-a00d-0339b9417aa7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:33.383399 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:33.383339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:33.383629 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:33.383431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:33.383629 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.383522 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:33.383629 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.383608 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret podName:79a6f67d-6571-4bbe-92e9-d640eb94ac19 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:41.383583851 +0000 UTC m=+18.153475424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret") pod "global-pull-secret-syncer-xrpmh" (UID: "79a6f67d-6571-4bbe-92e9-d640eb94ac19") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:33.383629 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.383609 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:33.383629 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.383628 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:33.383948 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.383641 2574 projected.go:196] Error preparing data for projected volume kube-api-access-lgf6k for pod openshift-network-diagnostics/network-check-target-hhtvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:33.383948 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.383691 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k podName:6b3ad089-857a-42f9-961f-42eb89b1a14b nodeName:}" failed. No retries permitted until 2026-07-06 16:38:41.383675409 +0000 UTC m=+18.153566930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgf6k" (UniqueName: "kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k") pod "network-check-target-hhtvk" (UID: "6b3ad089-857a-42f9-961f-42eb89b1a14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:33.744336 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:33.742758 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:33.744336 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.742881 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:33.744336 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:33.743280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:33.744336 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.743391 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:33.744336 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:33.743441 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:33.744336 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:33.743506 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:35.742552 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:35.742468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:35.743013 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:35.742614 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:35.743013 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:35.742473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:35.743013 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:35.742718 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:35.743013 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:35.742468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:35.743013 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:35.742790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:37.741957 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:37.741879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:37.741957 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:37.741899 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:37.741957 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:37.741915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:37.742484 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:37.742040 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:37.742484 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:37.742110 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:37.742484 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:37.742198 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:39.742914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:39.742878 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:39.743354 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:39.742878 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:39.743354 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:39.743008 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:39.743354 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:39.743061 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:39.743354 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:39.742889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:39.743354 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:39.743146 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:41.340856 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:41.340819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:41.341303 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.340984 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:41.341303 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.341058 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs podName:08928192-3dac-4b46-a00d-0339b9417aa7 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:57.341036683 +0000 UTC m=+34.110928220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs") pod "network-metrics-daemon-lhbj8" (UID: "08928192-3dac-4b46-a00d-0339b9417aa7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:41.441693 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:41.441645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:41.441874 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:41.441734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:41.441874 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.441818 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:41.441874 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.441850 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:41.441874 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.441866 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:41.441874 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.441878 2574 projected.go:196] Error preparing data for projected volume kube-api-access-lgf6k for pod openshift-network-diagnostics/network-check-target-hhtvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:41.442075 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.441889 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret podName:79a6f67d-6571-4bbe-92e9-d640eb94ac19 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:57.441871059 +0000 UTC m=+34.211762590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret") pod "global-pull-secret-syncer-xrpmh" (UID: "79a6f67d-6571-4bbe-92e9-d640eb94ac19") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:41.442075 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.441911 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k podName:6b3ad089-857a-42f9-961f-42eb89b1a14b nodeName:}" failed. No retries permitted until 2026-07-06 16:38:57.441900563 +0000 UTC m=+34.211792085 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgf6k" (UniqueName: "kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k") pod "network-check-target-hhtvk" (UID: "6b3ad089-857a-42f9-961f-42eb89b1a14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:41.742449 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:41.742415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:41.742641 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:41.742421 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:41.742641 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:41.742448 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:41.742641 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.742566 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:41.742792 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.742693 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:41.742792 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:41.742752 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:43.743354 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:43.743319 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:43.743848 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:43.743413 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:43.743848 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:43.743440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:43.743848 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:43.743488 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:43.743848 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:43.743529 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:43.743848 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:43.743604 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:45.742862 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:45.742824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:45.743392 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:45.742824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:45.743392 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:45.742942 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:45.743392 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:45.742969 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:45.743392 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:45.743075 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:45.743392 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:45.743154 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:46.833267 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.832839 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jpgx" event={"ID":"9b13ab05-cd28-4ff2-9b5f-92d2c511bff5","Type":"ContainerStarted","Data":"c5ac83e1cf605741938270995d26d4a9f7c2314fb6351bc6890401e6c2a7f09f"} Jul 06 16:38:46.834434 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.834404 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" event={"ID":"95ef54ee-5605-4f93-9324-0794af019db5","Type":"ContainerStarted","Data":"8eb3237e3b7111133b546de4af3409e234562f30212d54538bd6ab3c1af1a10c"} Jul 06 16:38:46.837254 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.837227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"9e711595cd6e44f6a176cbb5a81910279e4615e5ee3c4602b69af7ea56314db5"} Jul 06 16:38:46.837369 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.837300 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"5262791e791c75715c8faf27132ce8b8607340d282db9a1bc29e9075c3a991da"} Jul 06 16:38:46.837425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.837371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"6b9936e4084ca1aa353ae47065806f7ca3495172883eabd9580f88d9e764df21"} Jul 06 16:38:46.837425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.837409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"c3fa1c003b3636a70441a4d0aa741327bd190372013766db994776dca0a650cb"} Jul 06 16:38:46.837539 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.837441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"dad04018443a99d25d8d0978d363c7db3172d92c0101a22b08181c05dd6d49c9"} Jul 06 16:38:46.837539 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.837492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"309cbadb5a5d963221a8cd3429247f1c9b746f28d2be0dc063630dd79dbc01f1"} Jul 06 16:38:46.838568 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.838543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nflws" event={"ID":"c75e82f6-540f-4d90-be0a-ed9c9445152f","Type":"ContainerStarted","Data":"2d760e6a2efd5c9f1ac2ba6300c44cdc497d2d7d1d111ef3f462e2c14359d1b8"} Jul 06 16:38:46.839881 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.839859 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" event={"ID":"536cf0ac-af55-4f89-a588-3c6ebb380f18","Type":"ContainerStarted","Data":"72d57984b03baab7a91ebbdf5983f6f0ffaf339ce268b9d93ec0333c1936e467"} Jul 06 16:38:46.841111 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.841091 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pf8rk" event={"ID":"c5044fe4-50b7-4819-9820-07b80fab8014","Type":"ContainerStarted","Data":"e5e71e51a87f3bc45c7ae20cd9a6b8a5fc77e3f1616e6f5b9edd4d5e64ff0498"} Jul 06 16:38:46.842387 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.842366 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5w5l8" event={"ID":"3dcf852d-ac55-4a06-8597-f77c25cb65a5","Type":"ContainerStarted","Data":"df8acb21aae700ad3efabb30fd4e1aeeeecfee5574d74ab9e698dcfadb65245a"} Jul 06 16:38:46.843710 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.843691 2574 generic.go:356] "Generic (PLEG): container finished" podID="d48bd494-815a-4f2c-bd82-72cbbe63ca5c" containerID="3934cf53690eec0f4f565792fb4f3dc2c810b5eae9c386ffa4e3ed5c30553e5b" exitCode=0 Jul 06 16:38:46.843784 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.843750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" event={"ID":"d48bd494-815a-4f2c-bd82-72cbbe63ca5c","Type":"ContainerDied","Data":"3934cf53690eec0f4f565792fb4f3dc2c810b5eae9c386ffa4e3ed5c30553e5b"} Jul 06 16:38:46.845039 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.845013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" event={"ID":"1115f28ae51a4317d8862e47dd2c12eb","Type":"ContainerStarted","Data":"881a2c19a851ccb16f8f2aa6b28f966a39752184a65afa4a49bf3c4db33121f1"} Jul 06 16:38:46.869166 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.869117 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-239.ec2.internal" podStartSLOduration=21.869101632 podStartE2EDuration="21.869101632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:30.83348774 +0000 UTC m=+7.603379288" watchObservedRunningTime="2026-07-06 16:38:46.869101632 +0000 UTC m=+23.638993173" Jul 06 16:38:46.896701 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.896643 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8jpgx" podStartSLOduration=1.970936249 podStartE2EDuration="21.896620181s" totalImagesPullingTime="19.925683932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:46.869651012 +0000 UTC m=+23.639542554" watchObservedRunningTime="2026-07-06 16:38:46.896620181 +0000 UTC m=+23.666511721" Jul 06 16:38:46.922546 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.922496 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nflws" podStartSLOduration=2.961697756 podStartE2EDuration="22.922479634s" totalImagesPullingTime="19.960781878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:46.89673807 +0000 UTC m=+23.666629617" watchObservedRunningTime="2026-07-06 16:38:46.922479634 +0000 UTC m=+23.692371300" Jul 06 16:38:46.923276 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.923228 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5w5l8" podStartSLOduration=2.963633555 podStartE2EDuration="22.923213397s" totalImagesPullingTime="19.959579842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:46.92279738 +0000 UTC m=+23.692688922" watchObservedRunningTime="2026-07-06 16:38:46.923213397 +0000 UTC m=+23.693104941" Jul 06 16:38:46.952200 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:46.952143 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pf8rk" podStartSLOduration=2.944692513 podStartE2EDuration="22.952125482s" totalImagesPullingTime="20.007432969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:46.951880913 +0000 UTC m=+23.721772455" watchObservedRunningTime="2026-07-06 16:38:46.952125482 +0000 UTC m=+23.722017023" Jul 06 16:38:47.053200 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.053154 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-239.ec2.internal" podStartSLOduration=22.053140045 podStartE2EDuration="22.053140045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:46.999964907 +0000 UTC m=+23.769856447" watchObservedRunningTime="2026-07-06 16:38:47.053140045 +0000 UTC m=+23.823031646" Jul 06 16:38:47.096439 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.096341 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l9gc8" podStartSLOduration=3.157036896 podStartE2EDuration="23.096325222s" totalImagesPullingTime="19.939288326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:47.096047017 +0000 UTC m=+23.865938558" watchObservedRunningTime="2026-07-06 16:38:47.096325222 +0000 UTC m=+23.866216790" Jul 06 16:38:47.298396 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.298365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:47.415241 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.415202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:47.455332 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.455300 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 16:38:47.669009 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.668826 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T16:38:47.455324908Z","UUID":"0f6697c7-1a83-47b3-9956-302ab6c9d20d","Handler":null,"Name":"","Endpoint":""} Jul 06 16:38:47.670209 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.670192 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 16:38:47.670287 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.670229 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 16:38:47.742003 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.741963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:47.742003 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.741993 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:47.742228 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.741963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:47.742228 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:47.742092 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:47.742228 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:47.742179 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:47.742338 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:47.742249 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:47.848733 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.848695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r44dw" event={"ID":"04eeda10-18cb-4506-a057-e044524c5538","Type":"ContainerStarted","Data":"45de126897c6c7ab02d4fc8302ab8b8d3c75a3009f0a5eaf5cb869264ce1ea59"} Jul 06 16:38:47.850390 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:47.850357 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" event={"ID":"536cf0ac-af55-4f89-a588-3c6ebb380f18","Type":"ContainerStarted","Data":"9a5b9f7e2116d86e7bc66ce171cdc7261ff83068c9521f74c1e48f186e143ada"} Jul 06 16:38:48.133446 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:48.133361 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:48.233115 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:48.233078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:48.853747 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:48.853713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" event={"ID":"536cf0ac-af55-4f89-a588-3c6ebb380f18","Type":"ContainerStarted","Data":"83f721ee43f2fc792858b928965939771d08438d5dbef3d8516df2776fa93572"} Jul 06 16:38:48.856783 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:48.856755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"9115384b5bc50606745cca81cce58186407e548bc2047d82c587f7ff7e9631fd"} Jul 06 16:38:48.881867 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:48.881823 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r44dw" podStartSLOduration=12.460707686 podStartE2EDuration="23.881806333s" totalImagesPullingTime="11.421098647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:47.930077448 +0000 UTC m=+24.699968988" watchObservedRunningTime="2026-07-06 16:38:48.881806333 +0000 UTC m=+25.651697874" Jul 06 16:38:49.398378 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:49.398341 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:49.489983 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:49.489921 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:49.573592 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:49.573556 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:49.742266 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:49.742181 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:49.742266 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:49.742219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:49.742449 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:49.742181 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:49.742449 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:49.742290 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:49.742449 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:49.742344 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:49.742449 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:49.742410 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:50.359347 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:50.359313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:50.435772 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:50.435739 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:50.877024 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:50.876832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:50.958825 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:50.958758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:51.583141 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.583109 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:51.583849 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.583841 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:51.604397 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.604353 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mccp" podStartSLOduration=5.229999564 podStartE2EDuration="27.604334404s" totalImagesPullingTime="22.37433484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:48.882450811 +0000 UTC m=+25.652342351" watchObservedRunningTime="2026-07-06 16:38:51.604334404 +0000 UTC m=+28.374225923" Jul 06 16:38:51.742008 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.741920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:51.742008 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.741920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:51.742008 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.741920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:51.742232 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:51.742080 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:51.742285 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:51.742211 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:51.742368 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:51.742343 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:51.781805 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.781776 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jpgx_9b13ab05-cd28-4ff2-9b5f-92d2c511bff5/dns-node-resolver/0.log" Jul 06 16:38:51.867330 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.867282 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" event={"ID":"466a9c09-be51-4bee-a5d2-67fb2284c831","Type":"ContainerStarted","Data":"974dd54a512bfaa6bb0bdb8329b1326d4bdf839b93ec3ca1298be5af6dd01c7d"} Jul 06 16:38:51.867544 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.867469 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:51.867951 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.867931 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5w5l8" Jul 06 16:38:51.942190 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:51.942127 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" podStartSLOduration=7.901608983 podStartE2EDuration="27.942104s" totalImagesPullingTime="20.040495017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:51.916810781 +0000 UTC m=+28.686702322" watchObservedRunningTime="2026-07-06 16:38:51.942104 +0000 UTC m=+28.711995542" Jul 06 16:38:52.010367 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:52.010275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:52.783200 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:52.783032 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nflws_c75e82f6-540f-4d90-be0a-ed9c9445152f/node-ca/0.log" Jul 06 16:38:52.870063 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:52.870037 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:52.870327 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:52.870113 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:52.870327 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:52.870157 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:52.886153 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:52.886129 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:52.886299 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:52.886229 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:38:53.079385 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:53.079273 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:53.218185 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:53.218144 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:53.455082 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:53.454976 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hhtvk"] Jul 06 16:38:53.455236 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:53.455115 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:53.455236 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:53.455190 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:53.462972 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:53.462930 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xrpmh"] Jul 06 16:38:53.463148 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:53.463062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:53.463219 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:53.463173 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:53.470138 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:53.470103 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhbj8"] Jul 06 16:38:53.470312 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:53.470261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:53.470410 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:53.470386 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:55.741899 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:55.741861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:55.741899 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:55.741888 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:55.742447 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:55.741895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:55.742447 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:55.741988 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:55.742447 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:55.742104 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:55.742447 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:55.742189 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:56.686154 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:56.686064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:56.781231 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:56.781162 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:56.881317 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:56.881278 2574 generic.go:356] "Generic (PLEG): container finished" podID="d48bd494-815a-4f2c-bd82-72cbbe63ca5c" containerID="db44ccb913d0ce5f8c91b214d1b27ffa3b3ba210928fd6e4fc2d86761ebf20cc" exitCode=0 Jul 06 16:38:56.881479 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:56.881350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" event={"ID":"d48bd494-815a-4f2c-bd82-72cbbe63ca5c","Type":"ContainerDied","Data":"db44ccb913d0ce5f8c91b214d1b27ffa3b3ba210928fd6e4fc2d86761ebf20cc"} Jul 06 16:38:57.358311 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.358269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:57.358491 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.358419 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:57.358558 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.358502 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs podName:08928192-3dac-4b46-a00d-0339b9417aa7 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:29.358485608 +0000 UTC m=+66.128377130 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs") pod "network-metrics-daemon-lhbj8" (UID: "08928192-3dac-4b46-a00d-0339b9417aa7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:57.459364 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.459330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:57.459542 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.459379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:57.459542 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.459507 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:57.459542 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.459520 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:57.459542 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.459533 2574 projected.go:196] Error preparing data for projected volume kube-api-access-lgf6k for pod openshift-network-diagnostics/network-check-target-hhtvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:57.459542 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.459530 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:57.459711 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.459580 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k podName:6b3ad089-857a-42f9-961f-42eb89b1a14b nodeName:}" failed. No retries permitted until 2026-07-06 16:39:29.459567505 +0000 UTC m=+66.229459023 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgf6k" (UniqueName: "kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k") pod "network-check-target-hhtvk" (UID: "6b3ad089-857a-42f9-961f-42eb89b1a14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:57.459711 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.459604 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret podName:79a6f67d-6571-4bbe-92e9-d640eb94ac19 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:29.459589102 +0000 UTC m=+66.229480622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret") pod "global-pull-secret-syncer-xrpmh" (UID: "79a6f67d-6571-4bbe-92e9-d640eb94ac19") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:57.742450 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.742415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:57.742450 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.742415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:57.742678 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.742476 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:57.742678 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.742648 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhbj8" podUID="08928192-3dac-4b46-a00d-0339b9417aa7" Jul 06 16:38:57.742786 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.742762 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrpmh" podUID="79a6f67d-6571-4bbe-92e9-d640eb94ac19" Jul 06 16:38:57.742845 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:57.742830 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhtvk" podUID="6b3ad089-857a-42f9-961f-42eb89b1a14b" Jul 06 16:38:57.747793 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.747765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:57.885975 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.885939 2574 generic.go:356] "Generic (PLEG): container finished" podID="d48bd494-815a-4f2c-bd82-72cbbe63ca5c" containerID="9cabb24039f5f543cb248aeaf21caac6138e897c641bef572a6d135d4eeca2a7" exitCode=0 Jul 06 16:38:57.886317 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:57.885985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" event={"ID":"d48bd494-815a-4f2c-bd82-72cbbe63ca5c","Type":"ContainerDied","Data":"9cabb24039f5f543cb248aeaf21caac6138e897c641bef572a6d135d4eeca2a7"} Jul 06 16:38:58.890509 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:58.890403 2574 generic.go:356] "Generic (PLEG): container finished" podID="d48bd494-815a-4f2c-bd82-72cbbe63ca5c" containerID="7e0be7a14510cfd1e69463c29de2b8a397876ad30ea593a46d964f196b4774ec" exitCode=0 Jul 06 16:38:58.890509 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:58.890448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" event={"ID":"d48bd494-815a-4f2c-bd82-72cbbe63ca5c","Type":"ContainerDied","Data":"7e0be7a14510cfd1e69463c29de2b8a397876ad30ea593a46d964f196b4774ec"} Jul 06 16:38:59.495137 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.495109 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-239.ec2.internal" event="NodeReady" Jul 06 16:38:59.495285 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.495243 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 16:38:59.584911 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.584869 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dwrlf"] Jul 06 16:38:59.604784 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.604754 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4nrv9"] Jul 06 16:38:59.604967 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.604894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.609067 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.608887 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 16:38:59.609067 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.609012 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 16:38:59.609302 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.609166 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 16:38:59.609302 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.609242 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nl5jr\"" Jul 06 16:38:59.609530 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.609515 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 16:38:59.620733 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.620704 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4nrv9"] Jul 06 16:38:59.620733 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.620741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dwrlf"] Jul 06 16:38:59.620954 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.620833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4nrv9" Jul 06 16:38:59.624229 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.624204 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 16:38:59.624359 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.624322 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g8n7p\"" Jul 06 16:38:59.624940 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.624920 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 16:38:59.628396 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.628373 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 16:38:59.648110 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.648073 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-srdrb"] Jul 06 16:38:59.668766 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.668741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-srdrb"] Jul 06 16:38:59.668909 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.668873 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.673441 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.673415 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 16:38:59.673441 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.673438 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 16:38:59.673664 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.673511 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-db7nl\"" Jul 06 16:38:59.742627 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.742602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:38:59.742816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.742601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:38:59.742816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.742731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:38:59.746335 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.746103 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 16:38:59.746517 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.746366 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q4r89\"" Jul 06 16:38:59.746517 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.746409 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 16:38:59.746517 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.746438 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ntdtl\"" Jul 06 16:38:59.747872 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.747694 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 16:38:59.747872 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.747818 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 16:38:59.770331 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.770293 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:59.777283 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmllk\" (UniqueName: \"kubernetes.io/projected/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-kube-api-access-bmllk\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.777374 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-tmp-dir\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.777374 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86e55149-b176-417f-a475-35030db94939-crio-socket\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.777448 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmzfn\" (UniqueName: \"kubernetes.io/projected/86e55149-b176-417f-a475-35030db94939-kube-api-access-xmzfn\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.777448 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4cn\" (UniqueName: \"kubernetes.io/projected/720b190b-59b6-42bb-846c-73437ca1229d-kube-api-access-wb4cn\") pod \"ingress-canary-4nrv9\" (UID: \"720b190b-59b6-42bb-846c-73437ca1229d\") " pod="openshift-ingress-canary/ingress-canary-4nrv9" Jul 06 16:38:59.777448 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.777448 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e55149-b176-417f-a475-35030db94939-data-volume\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.777617 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777447 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.777617 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b190b-59b6-42bb-846c-73437ca1229d-cert\") pod \"ingress-canary-4nrv9\" (UID: \"720b190b-59b6-42bb-846c-73437ca1229d\") " pod="openshift-ingress-canary/ingress-canary-4nrv9" Jul 06 16:38:59.777617 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777589 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.777726 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.777638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-config-volume\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.877433 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.877388 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:38:59.878275 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-config-volume\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.878275 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmllk\" (UniqueName: \"kubernetes.io/projected/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-kube-api-access-bmllk\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.878476 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-tmp-dir\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.878476 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86e55149-b176-417f-a475-35030db94939-crio-socket\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.878476 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878408 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86e55149-b176-417f-a475-35030db94939-crio-socket\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.878476 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmzfn\" (UniqueName: \"kubernetes.io/projected/86e55149-b176-417f-a475-35030db94939-kube-api-access-xmzfn\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.878666 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4cn\" (UniqueName: \"kubernetes.io/projected/720b190b-59b6-42bb-846c-73437ca1229d-kube-api-access-wb4cn\") pod \"ingress-canary-4nrv9\" (UID: \"720b190b-59b6-42bb-846c-73437ca1229d\") " pod="openshift-ingress-canary/ingress-canary-4nrv9" Jul 06 16:38:59.878666 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.878666 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e55149-b176-417f-a475-35030db94939-data-volume\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.878666 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.878666 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b190b-59b6-42bb-846c-73437ca1229d-cert\") pod \"ingress-canary-4nrv9\" (UID: \"720b190b-59b6-42bb-846c-73437ca1229d\") " pod="openshift-ingress-canary/ingress-canary-4nrv9" Jul 06 16:38:59.878900 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.878900 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:59.878719 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:38:59.878900 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:59.878767 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:38:59.878900 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:59.878778 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:38:59.878900 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:59.878843 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:00.378823175 +0000 UTC m=+37.148714708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : secret "insights-runtime-extractor-tls" not found Jul 06 16:38:59.878900 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:59.878869 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:00.378855975 +0000 UTC m=+37.148747499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : configmap "kube-rbac-proxy" not found Jul 06 16:38:59.879110 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:38:59.878916 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls podName:130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:00.378896726 +0000 UTC m=+37.148788250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls") pod "dns-default-srdrb" (UID: "130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6") : secret "dns-default-metrics-tls" not found Jul 06 16:38:59.879110 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.878944 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-config-volume\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.879110 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.879049 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e55149-b176-417f-a475-35030db94939-data-volume\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.879110 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.879058 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-tmp-dir\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.883126 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.883106 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b190b-59b6-42bb-846c-73437ca1229d-cert\") pod \"ingress-canary-4nrv9\" (UID: \"720b190b-59b6-42bb-846c-73437ca1229d\") " pod="openshift-ingress-canary/ingress-canary-4nrv9" Jul 06 16:38:59.889842 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.889819 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmllk\" (UniqueName: \"kubernetes.io/projected/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-kube-api-access-bmllk\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:38:59.904723 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.904685 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4cn\" (UniqueName: \"kubernetes.io/projected/720b190b-59b6-42bb-846c-73437ca1229d-kube-api-access-wb4cn\") pod \"ingress-canary-4nrv9\" (UID: \"720b190b-59b6-42bb-846c-73437ca1229d\") " pod="openshift-ingress-canary/ingress-canary-4nrv9" Jul 06 16:38:59.905574 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.905551 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmzfn\" (UniqueName: \"kubernetes.io/projected/86e55149-b176-417f-a475-35030db94939-kube-api-access-xmzfn\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:38:59.932485 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.932435 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4nrv9" Jul 06 16:38:59.967929 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:38:59.967895 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:00.161491 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.161396 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4nrv9"] Jul 06 16:39:00.166579 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:00.166545 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720b190b_59b6_42bb_846c_73437ca1229d.slice/crio-781d45da83a32971335ae08c5c08cbbc0f699ff45097bc9ef3067496bc143ee5 WatchSource:0}: Error finding container 781d45da83a32971335ae08c5c08cbbc0f699ff45097bc9ef3067496bc143ee5: Status 404 returned error can't find the container with id 781d45da83a32971335ae08c5c08cbbc0f699ff45097bc9ef3067496bc143ee5 Jul 06 16:39:00.382469 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.382362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:00.382642 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.382486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:00.382642 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.382517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:00.382642 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:00.382548 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:39:00.382642 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:00.382633 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:01.382612652 +0000 UTC m=+38.152504174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : configmap "kube-rbac-proxy" not found Jul 06 16:39:00.382847 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:00.382643 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:00.382847 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:00.382709 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:01.382692976 +0000 UTC m=+38.152584512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:00.382847 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:00.382644 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:00.382847 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:00.382750 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls podName:130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:01.382740082 +0000 UTC m=+38.152631604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls") pod "dns-default-srdrb" (UID: "130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6") : secret "dns-default-metrics-tls" not found Jul 06 16:39:00.388391 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.388361 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s"] Jul 06 16:39:00.420285 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.420252 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s"] Jul 06 16:39:00.420486 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.420355 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s" Jul 06 16:39:00.428567 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.428536 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-phbkj\"" Jul 06 16:39:00.583881 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.583849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwv46\" (UniqueName: \"kubernetes.io/projected/e532fcc9-b861-40a7-a528-034cf6a7eff4-kube-api-access-xwv46\") pod \"network-check-source-989f8b8dc-x5z8s\" (UID: \"e532fcc9-b861-40a7-a528-034cf6a7eff4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s" Jul 06 16:39:00.684939 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.684846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwv46\" (UniqueName: \"kubernetes.io/projected/e532fcc9-b861-40a7-a528-034cf6a7eff4-kube-api-access-xwv46\") pod \"network-check-source-989f8b8dc-x5z8s\" (UID: \"e532fcc9-b861-40a7-a528-034cf6a7eff4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s" Jul 06 16:39:00.715649 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.715449 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwv46\" (UniqueName: \"kubernetes.io/projected/e532fcc9-b861-40a7-a528-034cf6a7eff4-kube-api-access-xwv46\") pod \"network-check-source-989f8b8dc-x5z8s\" (UID: \"e532fcc9-b861-40a7-a528-034cf6a7eff4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s" Jul 06 16:39:00.728818 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.728788 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:00.731534 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.731513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s" Jul 06 16:39:00.785591 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.785559 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl"] Jul 06 16:39:00.801682 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.801652 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" Jul 06 16:39:00.804539 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.804511 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl"] Jul 06 16:39:00.806287 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.806253 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-md8xl\"" Jul 06 16:39:00.807154 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.807134 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 16:39:00.807423 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.807406 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 16:39:00.850920 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.850753 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:00.886263 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.886235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cqv\" (UniqueName: \"kubernetes.io/projected/6be92524-de0c-4b62-9df8-28acd33362e7-kube-api-access-d7cqv\") pod \"migrator-d57d58cc9-csndl\" (UID: \"6be92524-de0c-4b62-9df8-28acd33362e7\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" Jul 06 16:39:00.896589 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.896552 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4nrv9" event={"ID":"720b190b-59b6-42bb-846c-73437ca1229d","Type":"ContainerStarted","Data":"781d45da83a32971335ae08c5c08cbbc0f699ff45097bc9ef3067496bc143ee5"} Jul 06 16:39:00.930642 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.930607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s"] Jul 06 16:39:00.934123 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:00.934094 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode532fcc9_b861_40a7_a528_034cf6a7eff4.slice/crio-fcd206dba21f48bf7ab94583e7a9ef37fe28bb5fb2b44f52d97db7c0400a29dd WatchSource:0}: Error finding container fcd206dba21f48bf7ab94583e7a9ef37fe28bb5fb2b44f52d97db7c0400a29dd: Status 404 returned error can't find the container with id fcd206dba21f48bf7ab94583e7a9ef37fe28bb5fb2b44f52d97db7c0400a29dd Jul 06 16:39:00.969636 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.969599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:00.987694 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:00.987659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cqv\" (UniqueName: \"kubernetes.io/projected/6be92524-de0c-4b62-9df8-28acd33362e7-kube-api-access-d7cqv\") pod \"migrator-d57d58cc9-csndl\" (UID: \"6be92524-de0c-4b62-9df8-28acd33362e7\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" Jul 06 16:39:01.007338 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.007307 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cqv\" (UniqueName: \"kubernetes.io/projected/6be92524-de0c-4b62-9df8-28acd33362e7-kube-api-access-d7cqv\") pod \"migrator-d57d58cc9-csndl\" (UID: \"6be92524-de0c-4b62-9df8-28acd33362e7\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" Jul 06 16:39:01.059251 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.059222 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b7c99dc67-wf7vr"] Jul 06 16:39:01.080013 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.079983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.083429 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.083404 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 16:39:01.083827 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.083804 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 16:39:01.083975 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.083959 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rjds\"" Jul 06 16:39:01.093350 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.093328 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7c99dc67-wf7vr"] Jul 06 16:39:01.101925 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.101898 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 16:39:01.118214 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.115425 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 16:39:01.118214 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.115800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" Jul 06 16:39:01.166585 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.166553 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:01.189571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.189482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-image-registry-private-configuration\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.189571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.189536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.189770 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.189586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-trusted-ca\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.189770 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.189639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-installation-pull-secrets\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.189770 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.189686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzqb\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-kube-api-access-5gzqb\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.189770 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.189716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8e96ca8-3169-4c50-a628-12122ac66d91-ca-trust-extracted\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.189926 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.189816 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-certificates\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.189926 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.189851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-bound-sa-token\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.287398 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.287233 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.290610 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-image-registry-private-configuration\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.290659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.290684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-trusted-ca\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.290738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-installation-pull-secrets\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.290772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzqb\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-kube-api-access-5gzqb\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.290802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8e96ca8-3169-4c50-a628-12122ac66d91-ca-trust-extracted\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.290838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-certificates\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.290863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-bound-sa-token\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.291582 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8e96ca8-3169-4c50-a628-12122ac66d91-ca-trust-extracted\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.291669 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.291684 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7c99dc67-wf7vr: secret "image-registry-tls" not found Jul 06 16:39:01.292281 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.291736 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls podName:d8e96ca8-3169-4c50-a628-12122ac66d91 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:01.791716233 +0000 UTC m=+38.561607763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls") pod "image-registry-6b7c99dc67-wf7vr" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91") : secret "image-registry-tls" not found Jul 06 16:39:01.293016 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.292879 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-trusted-ca\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.293275 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.293238 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-certificates\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.293943 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.293919 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-image-registry-private-configuration\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.294125 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.294080 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-installation-pull-secrets\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.333786 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.333750 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-bound-sa-token\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.335173 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.335140 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzqb\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-kube-api-access-5gzqb\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.346639 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.346606 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl"] Jul 06 16:39:01.350535 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:01.350503 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be92524_de0c_4b62_9df8_28acd33362e7.slice/crio-8164b8ba6d2fea2ce13aa4b87b9278c11284c86a467600ef23040d890445482a WatchSource:0}: Error finding container 8164b8ba6d2fea2ce13aa4b87b9278c11284c86a467600ef23040d890445482a: Status 404 returned error can't find the container with id 8164b8ba6d2fea2ce13aa4b87b9278c11284c86a467600ef23040d890445482a Jul 06 16:39:01.392098 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.392063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:01.392267 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.392201 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:39:01.392337 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.392291 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:01.392337 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.392313 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:03.392298033 +0000 UTC m=+40.162189557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : configmap "kube-rbac-proxy" not found Jul 06 16:39:01.392443 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.392343 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls podName:130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:03.392326804 +0000 UTC m=+40.162218347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls") pod "dns-default-srdrb" (UID: "130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6") : secret "dns-default-metrics-tls" not found Jul 06 16:39:01.392443 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.392200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:01.392443 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.392382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:01.392690 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.392477 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:01.392690 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.392515 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:03.39250197 +0000 UTC m=+40.162393491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:01.796423 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.796380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:01.796633 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.796539 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:39:01.796633 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.796560 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7c99dc67-wf7vr: secret "image-registry-tls" not found Jul 06 16:39:01.796633 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:01.796624 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls podName:d8e96ca8-3169-4c50-a628-12122ac66d91 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:02.79660667 +0000 UTC m=+39.566498212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls") pod "image-registry-6b7c99dc67-wf7vr" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91") : secret "image-registry-tls" not found Jul 06 16:39:01.888319 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.888281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:01.900603 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.900564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s" event={"ID":"e532fcc9-b861-40a7-a528-034cf6a7eff4","Type":"ContainerStarted","Data":"fcd206dba21f48bf7ab94583e7a9ef37fe28bb5fb2b44f52d97db7c0400a29dd"} Jul 06 16:39:01.902180 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:01.902148 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" event={"ID":"6be92524-de0c-4b62-9df8-28acd33362e7","Type":"ContainerStarted","Data":"8164b8ba6d2fea2ce13aa4b87b9278c11284c86a467600ef23040d890445482a"} Jul 06 16:39:02.007499 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:02.007409 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:02.150755 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:02.150646 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:02.807001 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:02.806959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:02.807216 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:02.807143 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:39:02.807216 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:02.807166 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7c99dc67-wf7vr: secret "image-registry-tls" not found Jul 06 16:39:02.807347 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:02.807241 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls podName:d8e96ca8-3169-4c50-a628-12122ac66d91 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:04.807217104 +0000 UTC m=+41.577108650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls") pod "image-registry-6b7c99dc67-wf7vr" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91") : secret "image-registry-tls" not found Jul 06 16:39:03.412468 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:03.412414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:03.412898 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:03.412499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:03.412898 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:03.412554 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:03.412898 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:03.412584 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:03.412898 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:03.412642 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:03.412898 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:03.412665 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:39:03.412898 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:03.412671 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls podName:130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:07.412649867 +0000 UTC m=+44.182541391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls") pod "dns-default-srdrb" (UID: "130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6") : secret "dns-default-metrics-tls" not found Jul 06 16:39:03.412898 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:03.412694 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:07.412681585 +0000 UTC m=+44.182573103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:03.412898 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:03.412711 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:07.412701972 +0000 UTC m=+44.182593493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : configmap "kube-rbac-proxy" not found Jul 06 16:39:03.568756 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:03.568711 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:03.666900 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:03.666814 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:04.823998 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:04.823962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:04.824371 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:04.824108 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:39:04.824371 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:04.824123 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7c99dc67-wf7vr: secret "image-registry-tls" not found Jul 06 16:39:04.824371 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:04.824175 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls podName:d8e96ca8-3169-4c50-a628-12122ac66d91 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:08.824160275 +0000 UTC m=+45.594051794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls") pod "image-registry-6b7c99dc67-wf7vr" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91") : secret "image-registry-tls" not found Jul 06 16:39:06.230587 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:06.230549 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:06.361705 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:06.361668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:06.906198 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:06.906155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:07.025767 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.025739 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:07.446408 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.446368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:07.446408 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.446407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:07.446957 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.446439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:07.446957 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:07.446542 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:07.446957 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:07.446606 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls podName:130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:15.446591173 +0000 UTC m=+52.216482696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls") pod "dns-default-srdrb" (UID: "130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6") : secret "dns-default-metrics-tls" not found Jul 06 16:39:07.446957 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:07.446543 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:07.446957 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:07.446641 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:15.446634838 +0000 UTC m=+52.216526356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:07.446957 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.446947 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e55149-b176-417f-a475-35030db94939-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:07.920634 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.920595 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4nrv9" event={"ID":"720b190b-59b6-42bb-846c-73437ca1229d","Type":"ContainerStarted","Data":"5a9816b2a9f96a83ebfbc73226133d88304695709638da2a2ab47422fbce4f2e"} Jul 06 16:39:07.922409 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.922380 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" event={"ID":"6be92524-de0c-4b62-9df8-28acd33362e7","Type":"ContainerStarted","Data":"5bccdd8625b78b73c2e9a669a815d66bd793c10595ccb97b2dd3a3d917c98279"} Jul 06 16:39:07.922580 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.922432 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" event={"ID":"6be92524-de0c-4b62-9df8-28acd33362e7","Type":"ContainerStarted","Data":"5fff5819fb91d60d2842810c08e9db7a7f4e1eb070ed932e0fb0e5243a0ee4dc"} Jul 06 16:39:07.923809 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.923782 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s" event={"ID":"e532fcc9-b861-40a7-a528-034cf6a7eff4","Type":"ContainerStarted","Data":"deb18a6e2efaffa530a4fbc3f9bb502edd8bab41eaec64dec40515bd0629c1e5"} Jul 06 16:39:07.926442 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.926417 2574 generic.go:356] "Generic (PLEG): container finished" podID="d48bd494-815a-4f2c-bd82-72cbbe63ca5c" containerID="f4c536a357ad7901533022c38847c05b875a7d381fcb31be187f288638d01fa5" exitCode=0 Jul 06 16:39:07.926540 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.926470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" event={"ID":"d48bd494-815a-4f2c-bd82-72cbbe63ca5c","Type":"ContainerDied","Data":"f4c536a357ad7901533022c38847c05b875a7d381fcb31be187f288638d01fa5"} Jul 06 16:39:07.949098 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:07.949049 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4nrv9" podStartSLOduration=2.175799579 podStartE2EDuration="8.949034882s" totalImagesPullingTime="6.773235303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:07.948776872 +0000 UTC m=+44.718668412" watchObservedRunningTime="2026-07-06 16:39:07.949034882 +0000 UTC m=+44.718926422" Jul 06 16:39:08.015474 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:08.015410 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-csndl" podStartSLOduration=2.426873571 podStartE2EDuration="8.015393418s" totalImagesPullingTime="5.588519847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:07.973770041 +0000 UTC m=+44.743661583" watchObservedRunningTime="2026-07-06 16:39:08.015393418 +0000 UTC m=+44.785285168" Jul 06 16:39:08.039221 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:08.039165 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-x5z8s" podStartSLOduration=2.023222938 podStartE2EDuration="8.039147082s" totalImagesPullingTime="6.015924144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:08.038424538 +0000 UTC m=+44.808316079" watchObservedRunningTime="2026-07-06 16:39:08.039147082 +0000 UTC m=+44.809038623" Jul 06 16:39:08.558576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:08.558536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:08.857755 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:08.857657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:08.857931 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:08.857848 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:39:08.857931 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:08.857874 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7c99dc67-wf7vr: secret "image-registry-tls" not found Jul 06 16:39:08.858027 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:08.857944 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls podName:d8e96ca8-3169-4c50-a628-12122ac66d91 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:16.857922727 +0000 UTC m=+53.627814276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls") pod "image-registry-6b7c99dc67-wf7vr" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91") : secret "image-registry-tls" not found Jul 06 16:39:08.932124 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:08.932090 2574 generic.go:356] "Generic (PLEG): container finished" podID="d48bd494-815a-4f2c-bd82-72cbbe63ca5c" containerID="fc2c1183a9b59e386310ff64ce40f15a06700eaf6e6817da092b6284d810ee03" exitCode=0 Jul 06 16:39:08.932310 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:08.932202 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" event={"ID":"d48bd494-815a-4f2c-bd82-72cbbe63ca5c","Type":"ContainerDied","Data":"fc2c1183a9b59e386310ff64ce40f15a06700eaf6e6817da092b6284d810ee03"} Jul 06 16:39:09.937951 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:09.937917 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" event={"ID":"d48bd494-815a-4f2c-bd82-72cbbe63ca5c","Type":"ContainerStarted","Data":"d3790c1e05537fc7e53a0194ac0699da76d34b8bee98b6962c90926355469771"} Jul 06 16:39:10.108147 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:10.108085 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sj9ll" podStartSLOduration=15.306966911 podStartE2EDuration="46.108062695s" totalImagesPullingTime="30.801095784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:10.046087345 +0000 UTC m=+46.815978885" watchObservedRunningTime="2026-07-06 16:39:10.108062695 +0000 UTC m=+46.877954237" Jul 06 16:39:11.721318 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:11.721276 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:11.972642 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:11.972542 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:12.072918 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:12.072882 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:12.241277 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:12.241238 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:14.066526 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:14.066487 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:14.170470 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:14.170417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:14.290296 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:14.290254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:14.423293 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:14.423201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:15.409128 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:15.409084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:15.510743 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:15.510703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:15.510743 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:15.510744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:15.511005 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:15.510864 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:15.511005 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:15.510935 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls podName:86e55149-b176-417f-a475-35030db94939 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:31.510915544 +0000 UTC m=+68.280807080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dwrlf" (UID: "86e55149-b176-417f-a475-35030db94939") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:15.513657 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:15.513636 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6-metrics-tls\") pod \"dns-default-srdrb\" (UID: \"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6\") " pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:15.577218 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:15.577179 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:15.750015 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:15.749986 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-srdrb"] Jul 06 16:39:15.753928 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:15.753896 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130b3c20_a6bf_4ed4_a3ee_220fd30a3cc6.slice/crio-8c7a0e6cd6ab342259eedf505820993856760211bf351018f2cc78f69f3a19de WatchSource:0}: Error finding container 8c7a0e6cd6ab342259eedf505820993856760211bf351018f2cc78f69f3a19de: Status 404 returned error can't find the container with id 8c7a0e6cd6ab342259eedf505820993856760211bf351018f2cc78f69f3a19de Jul 06 16:39:15.952750 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:15.952663 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-srdrb" event={"ID":"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6","Type":"ContainerStarted","Data":"8c7a0e6cd6ab342259eedf505820993856760211bf351018f2cc78f69f3a19de"} Jul 06 16:39:16.922888 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:16.922835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:16.925721 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:16.925691 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"image-registry-6b7c99dc67-wf7vr\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:16.990093 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:16.990054 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:17.168218 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:17.168184 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7c99dc67-wf7vr"] Jul 06 16:39:17.171817 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:17.171788 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e96ca8_3169_4c50_a628_12122ac66d91.slice/crio-8e299e78b2873dea3ab6a7fdbaa346c68f6a151bc01a90ddced618392d68358e WatchSource:0}: Error finding container 8e299e78b2873dea3ab6a7fdbaa346c68f6a151bc01a90ddced618392d68358e: Status 404 returned error can't find the container with id 8e299e78b2873dea3ab6a7fdbaa346c68f6a151bc01a90ddced618392d68358e Jul 06 16:39:17.317170 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:17.317140 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:17.959276 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:17.959237 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" event={"ID":"d8e96ca8-3169-4c50-a628-12122ac66d91","Type":"ContainerStarted","Data":"eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30"} Jul 06 16:39:17.959729 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:17.959297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" event={"ID":"d8e96ca8-3169-4c50-a628-12122ac66d91","Type":"ContainerStarted","Data":"8e299e78b2873dea3ab6a7fdbaa346c68f6a151bc01a90ddced618392d68358e"} Jul 06 16:39:17.959729 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:17.959398 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:18.005109 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:18.005048 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" podStartSLOduration=17.005026806 podStartE2EDuration="17.005026806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:18.004366172 +0000 UTC m=+54.774257715" watchObservedRunningTime="2026-07-06 16:39:18.005026806 +0000 UTC m=+54.774918341" Jul 06 16:39:18.964689 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:18.964599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-srdrb" event={"ID":"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6","Type":"ContainerStarted","Data":"91c125d6bc2a5ceacc8a49bbe7d067c2d2fb382b9419fd8be29989cc43da2467"} Jul 06 16:39:18.964689 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:18.964662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-srdrb" event={"ID":"130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6","Type":"ContainerStarted","Data":"8d2cb65c0943e0566321ec16819267377e9f1fc2f5a8547b2e367a9bf8695dc1"} Jul 06 16:39:19.115508 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:19.115469 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:19.234927 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:19.234890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:19.611627 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:19.611536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:19.721436 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:19.721385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:19.968426 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:19.968352 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:20.201946 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:20.201912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:20.343692 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:20.343664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:20.453054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:20.453022 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:20.627472 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:20.627362 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:20.811536 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:20.811506 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:21.797739 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:21.797704 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:22.205154 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:22.205064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:22.392803 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:22.392752 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:22.620096 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:22.620065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:22.806063 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:22.806028 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:24.534259 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.534208 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-srdrb" podStartSLOduration=22.74304303 podStartE2EDuration="25.534191184s" totalImagesPullingTime="2.791148154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:18.992386398 +0000 UTC m=+55.762277940" watchObservedRunningTime="2026-07-06 16:39:24.534191184 +0000 UTC m=+61.304082702" Jul 06 16:39:24.534654 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.534380 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp"] Jul 06 16:39:24.537061 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.537046 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" Jul 06 16:39:24.540815 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.540794 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 16:39:24.540979 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.540973 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q8qvd\"" Jul 06 16:39:24.544371 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.544349 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 16:39:24.544897 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.544882 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 16:39:24.547825 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.547812 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 16:39:24.558008 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.557988 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h"] Jul 06 16:39:24.559813 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.559798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.567814 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.567791 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 16:39:24.573269 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.573250 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b7c99dc67-wf7vr"] Jul 06 16:39:24.586914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.586891 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb903788-5bcc-4bc4-b221-aaed25458609-tmp\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.587048 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.586928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwpf\" (UniqueName: \"kubernetes.io/projected/cb903788-5bcc-4bc4-b221-aaed25458609-kube-api-access-hbwpf\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.587048 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.586990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wfl\" (UniqueName: \"kubernetes.io/projected/d1e768a2-b11a-4184-8a41-a850004716af-kube-api-access-s5wfl\") pod \"managed-serviceaccount-addon-agent-54d7dbd656-k4vtp\" (UID: \"d1e768a2-b11a-4184-8a41-a850004716af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" Jul 06 16:39:24.587048 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.587019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cb903788-5bcc-4bc4-b221-aaed25458609-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.587182 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.587047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1e768a2-b11a-4184-8a41-a850004716af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d7dbd656-k4vtp\" (UID: \"d1e768a2-b11a-4184-8a41-a850004716af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" Jul 06 16:39:24.588035 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.588019 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp"] Jul 06 16:39:24.655658 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.655626 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h"] Jul 06 16:39:24.687711 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.687672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb903788-5bcc-4bc4-b221-aaed25458609-tmp\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.687711 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.687716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwpf\" (UniqueName: \"kubernetes.io/projected/cb903788-5bcc-4bc4-b221-aaed25458609-kube-api-access-hbwpf\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.687969 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.687743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wfl\" (UniqueName: \"kubernetes.io/projected/d1e768a2-b11a-4184-8a41-a850004716af-kube-api-access-s5wfl\") pod \"managed-serviceaccount-addon-agent-54d7dbd656-k4vtp\" (UID: \"d1e768a2-b11a-4184-8a41-a850004716af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" Jul 06 16:39:24.687969 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.687767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cb903788-5bcc-4bc4-b221-aaed25458609-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.687969 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.687792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1e768a2-b11a-4184-8a41-a850004716af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d7dbd656-k4vtp\" (UID: \"d1e768a2-b11a-4184-8a41-a850004716af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" Jul 06 16:39:24.688200 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.688087 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb903788-5bcc-4bc4-b221-aaed25458609-tmp\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.690301 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.690275 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1e768a2-b11a-4184-8a41-a850004716af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d7dbd656-k4vtp\" (UID: \"d1e768a2-b11a-4184-8a41-a850004716af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" Jul 06 16:39:24.690503 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.690482 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cb903788-5bcc-4bc4-b221-aaed25458609-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.728944 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.728914 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:24.733710 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.733691 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wfl\" (UniqueName: \"kubernetes.io/projected/d1e768a2-b11a-4184-8a41-a850004716af-kube-api-access-s5wfl\") pod \"managed-serviceaccount-addon-agent-54d7dbd656-k4vtp\" (UID: \"d1e768a2-b11a-4184-8a41-a850004716af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" Jul 06 16:39:24.737809 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.737775 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwpf\" (UniqueName: \"kubernetes.io/projected/cb903788-5bcc-4bc4-b221-aaed25458609-kube-api-access-hbwpf\") pod \"klusterlet-addon-workmgr-6d6c6559-nvm6h\" (UID: \"cb903788-5bcc-4bc4-b221-aaed25458609\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.759431 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.759400 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-768bdd6975-2lgqz"] Jul 06 16:39:24.761416 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.761399 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.788675 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.788588 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-registry-tls\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.788675 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.788623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00a1230c-44c3-49a0-85d7-e111c5852469-trusted-ca\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.788675 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.788651 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00a1230c-44c3-49a0-85d7-e111c5852469-image-registry-private-configuration\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.788902 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.788735 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00a1230c-44c3-49a0-85d7-e111c5852469-registry-certificates\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.788902 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.788761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-bound-sa-token\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.788902 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.788784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00a1230c-44c3-49a0-85d7-e111c5852469-installation-pull-secrets\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.788902 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.788806 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlmf\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-kube-api-access-jnlmf\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.788902 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.788829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00a1230c-44c3-49a0-85d7-e111c5852469-ca-trust-extracted\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.793050 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.793018 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-768bdd6975-2lgqz"] Jul 06 16:39:24.859544 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.859505 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" Jul 06 16:39:24.868600 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.868576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:24.882625 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.882592 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:24.886129 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.886097 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvsnk" Jul 06 16:39:24.889268 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.889240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00a1230c-44c3-49a0-85d7-e111c5852469-trusted-ca\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.889402 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.889274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00a1230c-44c3-49a0-85d7-e111c5852469-image-registry-private-configuration\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.889402 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.889307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00a1230c-44c3-49a0-85d7-e111c5852469-registry-certificates\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.889402 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.889339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-bound-sa-token\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.889402 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.889358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00a1230c-44c3-49a0-85d7-e111c5852469-installation-pull-secrets\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.889402 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.889377 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlmf\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-kube-api-access-jnlmf\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.889670 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.889404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00a1230c-44c3-49a0-85d7-e111c5852469-ca-trust-extracted\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.889670 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.889514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-registry-tls\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.890413 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.890379 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00a1230c-44c3-49a0-85d7-e111c5852469-registry-certificates\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.890542 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.890493 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00a1230c-44c3-49a0-85d7-e111c5852469-trusted-ca\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.890542 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.890505 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00a1230c-44c3-49a0-85d7-e111c5852469-ca-trust-extracted\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.892043 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.892016 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-registry-tls\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.892261 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.892244 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00a1230c-44c3-49a0-85d7-e111c5852469-image-registry-private-configuration\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.892840 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.892820 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00a1230c-44c3-49a0-85d7-e111c5852469-installation-pull-secrets\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.917124 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.917059 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-bound-sa-token\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:24.945775 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:24.945429 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlmf\" (UniqueName: \"kubernetes.io/projected/00a1230c-44c3-49a0-85d7-e111c5852469-kube-api-access-jnlmf\") pod \"image-registry-768bdd6975-2lgqz\" (UID: \"00a1230c-44c3-49a0-85d7-e111c5852469\") " pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:25.056119 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.056030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:25.069919 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.069882 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:25.092868 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:25.092845 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb903788_5bcc_4bc4_b221_aaed25458609.slice/crio-ece0b271b808142b38c66e7b47048a8a57a28443f689914c1379c8b7191035d9 WatchSource:0}: Error finding container ece0b271b808142b38c66e7b47048a8a57a28443f689914c1379c8b7191035d9: Status 404 returned error can't find the container with id ece0b271b808142b38c66e7b47048a8a57a28443f689914c1379c8b7191035d9 Jul 06 16:39:25.114573 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.114526 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp"] Jul 06 16:39:25.115552 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.115526 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h"] Jul 06 16:39:25.117794 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:25.117763 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1e768a2_b11a_4184_8a41_a850004716af.slice/crio-63a366068ca4e1ea29730556d3bedb33769b62ad06e0e86e927da994f40381bc WatchSource:0}: Error finding container 63a366068ca4e1ea29730556d3bedb33769b62ad06e0e86e927da994f40381bc: Status 404 returned error can't find the container with id 63a366068ca4e1ea29730556d3bedb33769b62ad06e0e86e927da994f40381bc Jul 06 16:39:25.220135 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.220103 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:25.258747 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.258717 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-768bdd6975-2lgqz"] Jul 06 16:39:25.261898 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:25.261874 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a1230c_44c3_49a0_85d7_e111c5852469.slice/crio-1770a921c4f941cc4940225e008ecb6ac1567536f7e6afca010074c19fe81c4c WatchSource:0}: Error finding container 1770a921c4f941cc4940225e008ecb6ac1567536f7e6afca010074c19fe81c4c: Status 404 returned error can't find the container with id 1770a921c4f941cc4940225e008ecb6ac1567536f7e6afca010074c19fe81c4c Jul 06 16:39:25.394491 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.394444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:25.547640 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.547593 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:25.702887 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.702797 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:25.832975 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.832928 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:25.993691 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.993649 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" event={"ID":"00a1230c-44c3-49a0-85d7-e111c5852469","Type":"ContainerStarted","Data":"b08bec99bf74686450e0149d1699ccb2c54949328cc8bd563bac6d5214b209b1"} Jul 06 16:39:25.993882 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.993724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" event={"ID":"00a1230c-44c3-49a0-85d7-e111c5852469","Type":"ContainerStarted","Data":"1770a921c4f941cc4940225e008ecb6ac1567536f7e6afca010074c19fe81c4c"} Jul 06 16:39:25.993882 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.993803 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:25.995157 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.995131 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" event={"ID":"cb903788-5bcc-4bc4-b221-aaed25458609","Type":"ContainerStarted","Data":"ece0b271b808142b38c66e7b47048a8a57a28443f689914c1379c8b7191035d9"} Jul 06 16:39:25.996528 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:25.996504 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" event={"ID":"d1e768a2-b11a-4184-8a41-a850004716af","Type":"ContainerStarted","Data":"63a366068ca4e1ea29730556d3bedb33769b62ad06e0e86e927da994f40381bc"} Jul 06 16:39:26.059476 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:26.055856 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" podStartSLOduration=2.055835738 podStartE2EDuration="2.055835738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:26.055573066 +0000 UTC m=+62.825464608" watchObservedRunningTime="2026-07-06 16:39:26.055835738 +0000 UTC m=+62.825727280" Jul 06 16:39:27.459138 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:27.459099 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:27.552431 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:27.552380 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:28.006280 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:28.006243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" event={"ID":"d1e768a2-b11a-4184-8a41-a850004716af","Type":"ContainerStarted","Data":"9c43fbc5a966e56ba700ed25c2fdd1969726bbb6d1357b918deae23e1811db14"} Jul 06 16:39:28.033380 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:28.033312 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7dbd656-k4vtp" podStartSLOduration=1.263915286 podStartE2EDuration="4.033290078s" totalImagesPullingTime="2.769374792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:28.030869169 +0000 UTC m=+64.800760710" watchObservedRunningTime="2026-07-06 16:39:28.033290078 +0000 UTC m=+64.803181630" Jul 06 16:39:29.431058 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.431015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:39:29.434055 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.434032 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 16:39:29.443713 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.443682 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08928192-3dac-4b46-a00d-0339b9417aa7-metrics-certs\") pod \"network-metrics-daemon-lhbj8\" (UID: \"08928192-3dac-4b46-a00d-0339b9417aa7\") " pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:39:29.458531 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.458506 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ntdtl\"" Jul 06 16:39:29.466022 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.466002 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhbj8" Jul 06 16:39:29.532223 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.532191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:39:29.532323 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.532262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:39:29.534745 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.534722 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgf6k\" (UniqueName: \"kubernetes.io/projected/6b3ad089-857a-42f9-961f-42eb89b1a14b-kube-api-access-lgf6k\") pod \"network-check-target-hhtvk\" (UID: \"6b3ad089-857a-42f9-961f-42eb89b1a14b\") " pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:39:29.539216 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.539193 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 16:39:29.545278 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.545252 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6f67d-6571-4bbe-92e9-d640eb94ac19-original-pull-secret\") pod \"global-pull-secret-syncer-xrpmh\" (UID: \"79a6f67d-6571-4bbe-92e9-d640eb94ac19\") " pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:39:29.764795 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.764768 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q4r89\"" Jul 06 16:39:29.766920 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.766899 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrpmh" Jul 06 16:39:29.771639 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.771606 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:39:29.825365 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.825317 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhbj8"] Jul 06 16:39:29.842761 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:29.841978 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08928192_3dac_4b46_a00d_0339b9417aa7.slice/crio-1316cdd6e12fd1dbc8a89eb264210cdd6397038e8cc2674d062a346b7717e242 WatchSource:0}: Error finding container 1316cdd6e12fd1dbc8a89eb264210cdd6397038e8cc2674d062a346b7717e242: Status 404 returned error can't find the container with id 1316cdd6e12fd1dbc8a89eb264210cdd6397038e8cc2674d062a346b7717e242 Jul 06 16:39:29.956824 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.956647 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xrpmh"] Jul 06 16:39:29.959299 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:29.959274 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a6f67d_6571_4bbe_92e9_d640eb94ac19.slice/crio-9e5f7adb9ff350c9fccf356290684521aa2c1551f8ba6cbfeaf750bad03c44fe WatchSource:0}: Error finding container 9e5f7adb9ff350c9fccf356290684521aa2c1551f8ba6cbfeaf750bad03c44fe: Status 404 returned error can't find the container with id 9e5f7adb9ff350c9fccf356290684521aa2c1551f8ba6cbfeaf750bad03c44fe Jul 06 16:39:29.976676 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.976649 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-srdrb" Jul 06 16:39:29.976843 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:29.976748 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hhtvk"] Jul 06 16:39:29.979839 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:29.979809 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3ad089_857a_42f9_961f_42eb89b1a14b.slice/crio-0a9491f50f971ae35154f5d5657f37a32529c6bb48e173b8405fbc64c6053cba WatchSource:0}: Error finding container 0a9491f50f971ae35154f5d5657f37a32529c6bb48e173b8405fbc64c6053cba: Status 404 returned error can't find the container with id 0a9491f50f971ae35154f5d5657f37a32529c6bb48e173b8405fbc64c6053cba Jul 06 16:39:30.014321 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:30.014284 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhbj8" event={"ID":"08928192-3dac-4b46-a00d-0339b9417aa7","Type":"ContainerStarted","Data":"1316cdd6e12fd1dbc8a89eb264210cdd6397038e8cc2674d062a346b7717e242"} Jul 06 16:39:30.015443 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:30.015410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xrpmh" event={"ID":"79a6f67d-6571-4bbe-92e9-d640eb94ac19","Type":"ContainerStarted","Data":"9e5f7adb9ff350c9fccf356290684521aa2c1551f8ba6cbfeaf750bad03c44fe"} Jul 06 16:39:30.016853 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:30.016826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" event={"ID":"cb903788-5bcc-4bc4-b221-aaed25458609","Type":"ContainerStarted","Data":"45ff5549b11ed725afd1db2a510d20751e919beedad1e5e6d50797fd90b9ff6a"} Jul 06 16:39:30.017048 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:30.017031 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:30.017996 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:30.017962 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hhtvk" event={"ID":"6b3ad089-857a-42f9-961f-42eb89b1a14b","Type":"ContainerStarted","Data":"0a9491f50f971ae35154f5d5657f37a32529c6bb48e173b8405fbc64c6053cba"} Jul 06 16:39:30.018962 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:30.018944 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" Jul 06 16:39:30.058892 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:30.058831 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d6c6559-nvm6h" podStartSLOduration=1.458440161 podStartE2EDuration="6.058810303s" totalImagesPullingTime="4.600370142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:30.057500095 +0000 UTC m=+66.827391634" watchObservedRunningTime="2026-07-06 16:39:30.058810303 +0000 UTC m=+66.828701845" Jul 06 16:39:31.024026 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.023927 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hhtvk" event={"ID":"6b3ad089-857a-42f9-961f-42eb89b1a14b","Type":"ContainerStarted","Data":"1fa77e428cd1db7ae41227f18d770b5178b683dca99b9df0446863aaee20bf8e"} Jul 06 16:39:31.024619 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.024183 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:39:31.062098 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.062036 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hhtvk" podStartSLOduration=67.062012434 podStartE2EDuration="1m7.062012434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:31.059580504 +0000 UTC m=+67.829472109" watchObservedRunningTime="2026-07-06 16:39:31.062012434 +0000 UTC m=+67.831903977" Jul 06 16:39:31.221483 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.221374 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:31.406509 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.406475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:31.548654 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.548602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:31.551224 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.551198 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e55149-b176-417f-a475-35030db94939-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dwrlf\" (UID: \"86e55149-b176-417f-a475-35030db94939\") " pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:31.718504 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.718398 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nl5jr\"" Jul 06 16:39:31.725705 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.725682 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dwrlf" Jul 06 16:39:31.900402 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:31.900372 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dwrlf"] Jul 06 16:39:31.903703 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:31.903678 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e55149_b176_417f_a475_35030db94939.slice/crio-4aa533725d6d387be29b510052b1c2b8ea3f34c6ca1e4b40d4c2fb4e66c54eb0 WatchSource:0}: Error finding container 4aa533725d6d387be29b510052b1c2b8ea3f34c6ca1e4b40d4c2fb4e66c54eb0: Status 404 returned error can't find the container with id 4aa533725d6d387be29b510052b1c2b8ea3f34c6ca1e4b40d4c2fb4e66c54eb0 Jul 06 16:39:32.031420 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:32.031387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhbj8" event={"ID":"08928192-3dac-4b46-a00d-0339b9417aa7","Type":"ContainerStarted","Data":"8f06685a42f2ac1cd3ee36fd3be85b280ec16cdddc9474df55a1c5479dbfd36e"} Jul 06 16:39:32.031887 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:32.031443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhbj8" event={"ID":"08928192-3dac-4b46-a00d-0339b9417aa7","Type":"ContainerStarted","Data":"2bcab80e9016baacf6119ae279830ca89a2bc9cca16ba721b6aad524b95a2400"} Jul 06 16:39:32.032507 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:32.032484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dwrlf" event={"ID":"86e55149-b176-417f-a475-35030db94939","Type":"ContainerStarted","Data":"4aa533725d6d387be29b510052b1c2b8ea3f34c6ca1e4b40d4c2fb4e66c54eb0"} Jul 06 16:39:32.063500 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:32.063437 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lhbj8" podStartSLOduration=66.586037707 podStartE2EDuration="1m8.063421196s" totalImagesPullingTime="1.477383489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:32.059961186 +0000 UTC m=+68.829852739" watchObservedRunningTime="2026-07-06 16:39:32.063421196 +0000 UTC m=+68.833312737" Jul 06 16:39:32.244346 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:32.244299 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:32.347371 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:32.347220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:34.579591 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:34.579547 2574 patch_prober.go:28] interesting pod/image-registry-6b7c99dc67-wf7vr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 16:39:34.579980 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:34.579611 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" podUID="d8e96ca8-3169-4c50-a628-12122ac66d91" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:39:34.766121 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:34.766007 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:34.914579 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:34.914497 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:36.657379 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:36.657326 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:36.794343 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:36.794288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:38.058228 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:38.058190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xrpmh" event={"ID":"79a6f67d-6571-4bbe-92e9-d640eb94ac19","Type":"ContainerStarted","Data":"349eeed3377d73be70a909386367cbdecbc7a568013e9101eef3f284ecb3ebbe"} Jul 06 16:39:38.059810 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:38.059776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dwrlf" event={"ID":"86e55149-b176-417f-a475-35030db94939","Type":"ContainerStarted","Data":"12a93b13d8c4c981d19b517c76aa9ee3861d38c074ae05a9a4ad340e6ee38d72"} Jul 06 16:39:38.090938 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:38.090885 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xrpmh" podStartSLOduration=66.023839053 podStartE2EDuration="1m13.090868202s" totalImagesPullingTime="7.067029149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:38.089609478 +0000 UTC m=+74.859501014" watchObservedRunningTime="2026-07-06 16:39:38.090868202 +0000 UTC m=+74.860759743" Jul 06 16:39:39.334072 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:39.334034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:39.448895 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:39.448862 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:39.822190 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:39.822148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:41.075704 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:41.075671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dwrlf" event={"ID":"86e55149-b176-417f-a475-35030db94939","Type":"ContainerStarted","Data":"e51b1eab0da1097c8fa9de1f2d04069b7c2bbe223411cd2ff8e8025cdf91f322"} Jul 06 16:39:41.186607 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:41.186570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:42.850992 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:42.850938 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:42.985857 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:42.985817 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:44.089221 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:44.089184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dwrlf" event={"ID":"86e55149-b176-417f-a475-35030db94939","Type":"ContainerStarted","Data":"84a28396a2ae0ee7e5cfa08eb2c39436503ecb3f499415f2b57a46919c817a2e"} Jul 06 16:39:44.579075 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:44.579043 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:44.614642 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:44.614593 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dwrlf" podStartSLOduration=33.91039124 podStartE2EDuration="45.614570237s" totalImagesPullingTime="11.704178997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:44.142118764 +0000 UTC m=+80.912010296" watchObservedRunningTime="2026-07-06 16:39:44.614570237 +0000 UTC m=+81.384461778" Jul 06 16:39:47.005091 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:47.004875 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-768bdd6975-2lgqz" Jul 06 16:39:47.324275 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:47.324193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:47.445183 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:47.445149 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:48.292051 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:48.292012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:48.395628 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:48.395599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:49.593786 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.593748 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" podUID="d8e96ca8-3169-4c50-a628-12122ac66d91" containerName="registry" containerID="cri-o://eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30" gracePeriod=30 Jul 06 16:39:49.830107 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.830082 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:49.892446 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.892359 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-installation-pull-secrets\") pod \"d8e96ca8-3169-4c50-a628-12122ac66d91\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " Jul 06 16:39:49.892446 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.892406 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-trusted-ca\") pod \"d8e96ca8-3169-4c50-a628-12122ac66d91\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " Jul 06 16:39:49.892446 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.892431 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-image-registry-private-configuration\") pod \"d8e96ca8-3169-4c50-a628-12122ac66d91\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " Jul 06 16:39:49.892756 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.892448 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8e96ca8-3169-4c50-a628-12122ac66d91-ca-trust-extracted\") pod \"d8e96ca8-3169-4c50-a628-12122ac66d91\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " Jul 06 16:39:49.892756 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.892499 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-bound-sa-token\") pod \"d8e96ca8-3169-4c50-a628-12122ac66d91\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " Jul 06 16:39:49.892756 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.892522 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gzqb\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-kube-api-access-5gzqb\") pod \"d8e96ca8-3169-4c50-a628-12122ac66d91\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " Jul 06 16:39:49.892756 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.892562 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-certificates\") pod \"d8e96ca8-3169-4c50-a628-12122ac66d91\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " Jul 06 16:39:49.892756 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.892602 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") pod \"d8e96ca8-3169-4c50-a628-12122ac66d91\" (UID: \"d8e96ca8-3169-4c50-a628-12122ac66d91\") " Jul 06 16:39:49.893083 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.893044 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d8e96ca8-3169-4c50-a628-12122ac66d91" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:39:49.893628 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.893595 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d8e96ca8-3169-4c50-a628-12122ac66d91" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:39:49.895078 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.895045 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d8e96ca8-3169-4c50-a628-12122ac66d91" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:39:49.895179 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.895145 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d8e96ca8-3169-4c50-a628-12122ac66d91" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:39:49.895248 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.895193 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-kube-api-access-5gzqb" (OuterVolumeSpecName: "kube-api-access-5gzqb") pod "d8e96ca8-3169-4c50-a628-12122ac66d91" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91"). InnerVolumeSpecName "kube-api-access-5gzqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:39:49.895306 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.895249 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d8e96ca8-3169-4c50-a628-12122ac66d91" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:39:49.895360 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.895338 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d8e96ca8-3169-4c50-a628-12122ac66d91" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:39:49.901372 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.901341 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e96ca8-3169-4c50-a628-12122ac66d91-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d8e96ca8-3169-4c50-a628-12122ac66d91" (UID: "d8e96ca8-3169-4c50-a628-12122ac66d91"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:39:49.993352 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.993311 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8e96ca8-3169-4c50-a628-12122ac66d91-ca-trust-extracted\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:39:49.993352 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.993343 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-bound-sa-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:39:49.993352 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.993352 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gzqb\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-kube-api-access-5gzqb\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:39:49.993352 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.993361 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-certificates\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:39:49.993637 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.993370 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8e96ca8-3169-4c50-a628-12122ac66d91-registry-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:39:49.993637 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.993380 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-installation-pull-secrets\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:39:49.993637 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.993388 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8e96ca8-3169-4c50-a628-12122ac66d91-trusted-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:39:49.993637 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:49.993402 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8e96ca8-3169-4c50-a628-12122ac66d91-image-registry-private-configuration\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:39:50.109853 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.109812 2574 generic.go:356] "Generic (PLEG): container finished" podID="d8e96ca8-3169-4c50-a628-12122ac66d91" containerID="eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30" exitCode=0 Jul 06 16:39:50.109853 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.109857 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" Jul 06 16:39:50.110044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.109878 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" event={"ID":"d8e96ca8-3169-4c50-a628-12122ac66d91","Type":"ContainerDied","Data":"eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30"} Jul 06 16:39:50.110044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.109924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7c99dc67-wf7vr" event={"ID":"d8e96ca8-3169-4c50-a628-12122ac66d91","Type":"ContainerDied","Data":"8e299e78b2873dea3ab6a7fdbaa346c68f6a151bc01a90ddced618392d68358e"} Jul 06 16:39:50.110044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.109946 2574 scope.go:117] "RemoveContainer" containerID="eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30" Jul 06 16:39:50.113368 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.113340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:50.118711 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.118691 2574 scope.go:117] "RemoveContainer" containerID="eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30" Jul 06 16:39:50.119006 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:39:50.118987 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30\": container with ID starting with eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30 not found: ID does not exist" containerID="eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30" Jul 06 16:39:50.119065 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.119015 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30"} err="failed to get container status \"eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30\": rpc error: code = NotFound desc = could not find container \"eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30\": container with ID starting with eac0f46e88ea4ef1e4c0b90e5c1dc27bec3a7211f50b38407e88f4b77026eb30 not found: ID does not exist" Jul 06 16:39:50.145349 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.145262 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b7c99dc67-wf7vr"] Jul 06 16:39:50.159143 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.159110 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b7c99dc67-wf7vr"] Jul 06 16:39:50.205221 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.205184 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9b8ddff5-8bfqm"] Jul 06 16:39:50.205552 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.205536 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8e96ca8-3169-4c50-a628-12122ac66d91" containerName="registry" Jul 06 16:39:50.205644 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.205554 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8e96ca8-3169-4c50-a628-12122ac66d91" containerName="registry" Jul 06 16:39:50.205644 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.205621 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8e96ca8-3169-4c50-a628-12122ac66d91" containerName="registry" Jul 06 16:39:50.209278 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.209254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.213437 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.213414 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 16:39:50.213586 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.213444 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 16:39:50.213667 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.213654 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 16:39:50.214450 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.214429 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 16:39:50.214553 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.214467 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 16:39:50.214553 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.214431 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-klg6g\"" Jul 06 16:39:50.214738 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.214723 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 16:39:50.214812 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.214797 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 16:39:50.238579 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.238536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9b8ddff5-8bfqm"] Jul 06 16:39:50.239283 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.239249 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:50.295222 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.295182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-oauth-config\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.295377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.295230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-config\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.295377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.295261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpl7q\" (UniqueName: \"kubernetes.io/projected/b3fced7a-1b73-4d4f-bb0b-23515519935e-kube-api-access-zpl7q\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.295377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.295286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-serving-cert\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.295377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.295316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-oauth-serving-cert\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.295377 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.295341 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-service-ca\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.396114 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-oauth-config\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.396114 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-config\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.396114 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpl7q\" (UniqueName: \"kubernetes.io/projected/b3fced7a-1b73-4d4f-bb0b-23515519935e-kube-api-access-zpl7q\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.396293 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-serving-cert\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.396293 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-oauth-serving-cert\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.396293 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-service-ca\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.396894 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396872 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-config\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.396894 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396880 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-service-ca\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.397031 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.396950 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-oauth-serving-cert\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.398656 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.398635 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-oauth-config\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.398727 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.398704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-serving-cert\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.408509 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.408487 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpl7q\" (UniqueName: \"kubernetes.io/projected/b3fced7a-1b73-4d4f-bb0b-23515519935e-kube-api-access-zpl7q\") pod \"console-5d9b8ddff5-8bfqm\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.518625 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.518594 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:39:50.675922 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.675846 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9b8ddff5-8bfqm"] Jul 06 16:39:50.678980 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:39:50.678955 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fced7a_1b73_4d4f_bb0b_23515519935e.slice/crio-a5e2dc4cdea0a21ebeb8d98e5e2e72f085424c2c78a108dad691ab6e22d07f54 WatchSource:0}: Error finding container a5e2dc4cdea0a21ebeb8d98e5e2e72f085424c2c78a108dad691ab6e22d07f54: Status 404 returned error can't find the container with id a5e2dc4cdea0a21ebeb8d98e5e2e72f085424c2c78a108dad691ab6e22d07f54 Jul 06 16:39:50.815300 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.815264 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:50.972952 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:50.972858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:51.115688 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:51.115657 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9b8ddff5-8bfqm" event={"ID":"b3fced7a-1b73-4d4f-bb0b-23515519935e","Type":"ContainerStarted","Data":"a5e2dc4cdea0a21ebeb8d98e5e2e72f085424c2c78a108dad691ab6e22d07f54"} Jul 06 16:39:51.450666 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:51.450623 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:51.532262 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:51.532230 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:51.748799 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:51.748761 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8e96ca8-3169-4c50-a628-12122ac66d91" path="/var/lib/kubelet/pods/d8e96ca8-3169-4c50-a628-12122ac66d91/volumes" Jul 06 16:39:52.068289 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:52.068156 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:52.164241 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:52.164202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:52.800865 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:52.800825 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:52.901004 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:52.900963 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:53.127155 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:53.127013 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:53.217782 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:53.217744 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:53.362368 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:53.362311 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:54.337964 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:54.337924 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:55.131502 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:55.131448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9b8ddff5-8bfqm" event={"ID":"b3fced7a-1b73-4d4f-bb0b-23515519935e","Type":"ContainerStarted","Data":"8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360"} Jul 06 16:39:55.168533 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:55.168480 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d9b8ddff5-8bfqm" podStartSLOduration=1.5825928299999998 podStartE2EDuration="5.16844374s" totalImagesPullingTime="3.58585091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:55.168186296 +0000 UTC m=+91.938077837" watchObservedRunningTime="2026-07-06 16:39:55.16844374 +0000 UTC m=+91.938335280" Jul 06 16:39:57.567369 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:57.567330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:57.707023 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:57.706985 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:57.847176 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:57.847091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:39:58.099058 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:58.098976 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d9b8ddff5-8bfqm_b3fced7a-1b73-4d4f-bb0b-23515519935e/console/0.log" Jul 06 16:39:58.891554 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:39:58.891519 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4nrv9_720b190b-59b6-42bb-846c-73437ca1229d/serve-healthcheck-canary/0.log" Jul 06 16:40:00.157646 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:00.157608 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:00.519686 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:00.519639 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:40:00.519890 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:00.519715 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:40:00.524763 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:00.524737 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:40:01.158936 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:01.158906 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:40:01.233658 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:01.233626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:01.454352 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:01.454270 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:01.663676 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:01.663645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:01.965959 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:01.965927 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:02.035408 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:02.035378 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hhtvk" Jul 06 16:40:02.102680 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:02.102638 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:02.570102 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:02.570067 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9b8ddff5-8bfqm"] Jul 06 16:40:02.832756 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:02.832666 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:03.080922 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.080886 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:03.196054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.195972 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:03.344678 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.344632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:03.571337 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.571309 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:03.690156 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.690118 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v"] Jul 06 16:40:03.719376 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.719339 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.729972 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.729947 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f6q9f\"" Jul 06 16:40:03.730201 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.730175 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 16:40:03.730265 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.730235 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 16:40:03.730313 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.730269 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 16:40:03.730313 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.730234 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 16:40:03.730382 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.730343 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 16:40:03.749851 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.749817 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:03.751353 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.751319 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v"] Jul 06 16:40:03.802749 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.802710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.802749 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.802751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.802973 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.802811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.802973 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.802846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbpzd\" (UniqueName: \"kubernetes.io/projected/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-kube-api-access-fbpzd\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.813553 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.813521 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qxkpx"] Jul 06 16:40:03.835124 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.835048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:03.904117 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.904078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.904117 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.904119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.904378 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.904152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.904378 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.904279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpzd\" (UniqueName: \"kubernetes.io/projected/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-kube-api-access-fbpzd\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.904964 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.904940 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.906681 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.906662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.906681 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.906673 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:03.925991 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.925959 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 16:40:03.937891 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.937863 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 16:40:03.950385 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.950314 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 16:40:03.950385 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.950314 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-64ncg\"" Jul 06 16:40:03.953016 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:03.952996 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpzd\" (UniqueName: \"kubernetes.io/projected/eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a-kube-api-access-fbpzd\") pod \"openshift-state-metrics-54cf96dfcf-vgx4v\" (UID: \"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:04.005747 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.005707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-metrics-client-ca\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.005747 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.005754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-wtmp\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.005976 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.005771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-tls\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.005976 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.005792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-root\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.005976 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.005845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-textfile\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.005976 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.005902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6c5h\" (UniqueName: \"kubernetes.io/projected/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-kube-api-access-t6c5h\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.005976 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.005965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-accelerators-collector-config\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.006130 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.006009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-sys\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.006130 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.006038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.027654 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.027624 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" Jul 06 16:40:04.071391 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.071360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:04.106563 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-metrics-client-ca\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.106768 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-wtmp\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.106768 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-tls\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.106887 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106771 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-wtmp\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.106887 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-root\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.106887 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-textfile\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.106887 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6c5h\" (UniqueName: \"kubernetes.io/projected/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-kube-api-access-t6c5h\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.107086 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-accelerators-collector-config\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.107086 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.106962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-sys\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.107086 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.107014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.107250 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.107225 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-textfile\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.107380 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.107361 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-metrics-client-ca\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.107436 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.107401 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-sys\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.107517 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.107318 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-root\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.108274 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.108181 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-accelerators-collector-config\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.109826 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.109807 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-tls\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.110408 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.110367 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.147487 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.147439 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6c5h\" (UniqueName: \"kubernetes.io/projected/eb01898e-d0ac-49f4-ba1a-b33e9b4a2422-kube-api-access-t6c5h\") pod \"node-exporter-qxkpx\" (UID: \"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422\") " pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.300854 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.300822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v"] Jul 06 16:40:04.304583 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:40:04.304556 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeec9a9b_b9f7_430a_a5d5_e23ffec7f44a.slice/crio-3acca0366b20d2c67a9ea5f5fb04926b8b4cecbf7fa9c294e798ac981431f66c WatchSource:0}: Error finding container 3acca0366b20d2c67a9ea5f5fb04926b8b4cecbf7fa9c294e798ac981431f66c: Status 404 returned error can't find the container with id 3acca0366b20d2c67a9ea5f5fb04926b8b4cecbf7fa9c294e798ac981431f66c Jul 06 16:40:04.443990 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.443959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qxkpx" Jul 06 16:40:04.464316 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.464289 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:04.476604 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:40:04.476575 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb01898e_d0ac_49f4_ba1a_b33e9b4a2422.slice/crio-a1249df59db2d4836add6446e119fa1e8832118c33e3635173adbaa298f4e527 WatchSource:0}: Error finding container a1249df59db2d4836add6446e119fa1e8832118c33e3635173adbaa298f4e527: Status 404 returned error can't find the container with id a1249df59db2d4836add6446e119fa1e8832118c33e3635173adbaa298f4e527 Jul 06 16:40:04.754093 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.754057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:04.954630 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:04.954590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:05.170714 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.170630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" event={"ID":"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a","Type":"ContainerStarted","Data":"291d4f535e55dde18f8b7900632aeb50a6b414d162fdd0b80504454d9f9d007b"} Jul 06 16:40:05.170714 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.170692 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" event={"ID":"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a","Type":"ContainerStarted","Data":"3c0c65756209fd17c5690a0d99e8bd4d50b24d013aac84131377e6480c6e2720"} Jul 06 16:40:05.170714 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.170713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" event={"ID":"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a","Type":"ContainerStarted","Data":"3acca0366b20d2c67a9ea5f5fb04926b8b4cecbf7fa9c294e798ac981431f66c"} Jul 06 16:40:05.171755 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.171728 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qxkpx" event={"ID":"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422","Type":"ContainerStarted","Data":"a1249df59db2d4836add6446e119fa1e8832118c33e3635173adbaa298f4e527"} Jul 06 16:40:05.311750 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.311713 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:05.349730 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.349700 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 16:40:05.366874 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.366842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.397269 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397238 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d8djn\"" Jul 06 16:40:05.397269 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397265 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 16:40:05.397550 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397313 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 16:40:05.397550 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397238 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 16:40:05.397550 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397546 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 16:40:05.397716 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397612 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 16:40:05.397716 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397655 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 16:40:05.397716 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397680 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 16:40:05.397716 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.397709 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 16:40:05.409621 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.409594 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 16:40:05.495126 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.495092 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 16:40:05.520429 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1f505df-8e0b-41ce-9564-dccd86f560a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520429 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520424 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1f505df-8e0b-41ce-9564-dccd86f560a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520690 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520690 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1f505df-8e0b-41ce-9564-dccd86f560a5-config-out\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520690 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-web-config\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520690 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520690 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1f505df-8e0b-41ce-9564-dccd86f560a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520690 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520690 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520969 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520969 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520969 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1f505df-8e0b-41ce-9564-dccd86f560a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.520969 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.520885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cg29\" (UniqueName: \"kubernetes.io/projected/d1f505df-8e0b-41ce-9564-dccd86f560a5-kube-api-access-7cg29\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622026 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.621985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622026 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1f505df-8e0b-41ce-9564-dccd86f560a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622274 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cg29\" (UniqueName: \"kubernetes.io/projected/d1f505df-8e0b-41ce-9564-dccd86f560a5-kube-api-access-7cg29\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622274 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1f505df-8e0b-41ce-9564-dccd86f560a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622370 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1f505df-8e0b-41ce-9564-dccd86f560a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622370 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622370 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1f505df-8e0b-41ce-9564-dccd86f560a5-config-out\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-web-config\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1f505df-8e0b-41ce-9564-dccd86f560a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.622865 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.622714 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1f505df-8e0b-41ce-9564-dccd86f560a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.623726 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.623253 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1f505df-8e0b-41ce-9564-dccd86f560a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.625310 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.625283 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.625411 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.625351 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1f505df-8e0b-41ce-9564-dccd86f560a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.625680 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.625652 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1f505df-8e0b-41ce-9564-dccd86f560a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.625867 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.625825 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.625951 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.625885 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.626011 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.625944 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.626316 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.626301 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1f505df-8e0b-41ce-9564-dccd86f560a5-config-out\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.626494 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.626476 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-web-config\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.627136 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.627116 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.627637 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.627615 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1f505df-8e0b-41ce-9564-dccd86f560a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.661583 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.661553 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cg29\" (UniqueName: \"kubernetes.io/projected/d1f505df-8e0b-41ce-9564-dccd86f560a5-kube-api-access-7cg29\") pod \"alertmanager-main-0\" (UID: \"d1f505df-8e0b-41ce-9564-dccd86f560a5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.676387 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.676357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 16:40:05.717813 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.717780 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:05.974995 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:05.974961 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 16:40:05.981189 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:40:05.981161 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f505df_8e0b_41ce_9564_dccd86f560a5.slice/crio-857a24e9461b8bb3b774fd36da922b90e2969fa0c44adc2b8c4cb398ebf35e82 WatchSource:0}: Error finding container 857a24e9461b8bb3b774fd36da922b90e2969fa0c44adc2b8c4cb398ebf35e82: Status 404 returned error can't find the container with id 857a24e9461b8bb3b774fd36da922b90e2969fa0c44adc2b8c4cb398ebf35e82 Jul 06 16:40:06.178181 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:06.178126 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qxkpx" event={"ID":"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422","Type":"ContainerStarted","Data":"619c37534bd47c8807dabe9468c10e693da025d93dc54eecbf879baca836f9bd"} Jul 06 16:40:06.179473 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:06.179420 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1f505df-8e0b-41ce-9564-dccd86f560a5","Type":"ContainerStarted","Data":"857a24e9461b8bb3b774fd36da922b90e2969fa0c44adc2b8c4cb398ebf35e82"} Jul 06 16:40:07.185075 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:07.185043 2574 generic.go:356] "Generic (PLEG): container finished" podID="eb01898e-d0ac-49f4-ba1a-b33e9b4a2422" containerID="619c37534bd47c8807dabe9468c10e693da025d93dc54eecbf879baca836f9bd" exitCode=0 Jul 06 16:40:07.185527 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:07.185151 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qxkpx" event={"ID":"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422","Type":"ContainerDied","Data":"619c37534bd47c8807dabe9468c10e693da025d93dc54eecbf879baca836f9bd"} Jul 06 16:40:07.187868 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:07.187839 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" event={"ID":"eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a","Type":"ContainerStarted","Data":"2dfef55800febf20990cdcd17d282d30f26bd9a36554ee3531dedf8e8df1338c"} Jul 06 16:40:07.305851 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:07.305798 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-vgx4v" podStartSLOduration=1.768333713 podStartE2EDuration="4.305780828s" totalImagesPullingTime="2.537447115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:40:07.304817657 +0000 UTC m=+104.074709211" watchObservedRunningTime="2026-07-06 16:40:07.305780828 +0000 UTC m=+104.075672370" Jul 06 16:40:08.160160 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:08.160117 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:08.194473 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:08.194427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qxkpx" event={"ID":"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422","Type":"ContainerStarted","Data":"6037d7f8cc5c14215134870535f20d8e970231fcce2c4e7b896bebfdacfa7762"} Jul 06 16:40:08.194867 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:08.194502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qxkpx" event={"ID":"eb01898e-d0ac-49f4-ba1a-b33e9b4a2422","Type":"ContainerStarted","Data":"10abd2f8973eb9d0a584634e2afef7dd758e5fba417053e54a0f368005b951d9"} Jul 06 16:40:08.350152 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:08.350100 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:08.520513 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:08.520478 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:09.001527 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.001445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:09.001967 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.001921 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qxkpx" podStartSLOduration=4.3514531 podStartE2EDuration="6.001904769s" totalImagesPullingTime="1.527105941s" totalInitContainerRuntime="123.345728ms" isStatefulPod=true podCreationTimestamp="2026-07-06 16:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:40:08.364292985 +0000 UTC m=+105.134184526" watchObservedRunningTime="2026-07-06 16:40:09.001904769 +0000 UTC m=+105.771796309" Jul 06 16:40:09.003173 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.003148 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66b9d4f887-5xftb"] Jul 06 16:40:09.048675 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.048638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.065976 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.065946 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 16:40:09.065976 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.065964 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 16:40:09.066215 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.066021 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g9665\"" Jul 06 16:40:09.066409 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.066391 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 16:40:09.066510 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.066492 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 16:40:09.066577 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.066506 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4469i7am20kef\"" Jul 06 16:40:09.119055 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.119024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66b9d4f887-5xftb"] Jul 06 16:40:09.150632 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.150597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-secret-metrics-server-tls\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.150804 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.150652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d7e15d77-2583-4602-b954-7ebee2216a26-audit-log\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.150804 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.150739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-secret-metrics-server-client-certs\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.150881 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.150810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-client-ca-bundle\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.150881 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.150853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6lc2\" (UniqueName: \"kubernetes.io/projected/d7e15d77-2583-4602-b954-7ebee2216a26-kube-api-access-b6lc2\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.150950 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.150903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d7e15d77-2583-4602-b954-7ebee2216a26-metrics-server-audit-profiles\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.150950 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.150927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e15d77-2583-4602-b954-7ebee2216a26-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.199398 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.199356 2574 generic.go:356] "Generic (PLEG): container finished" podID="d1f505df-8e0b-41ce-9564-dccd86f560a5" containerID="365d40ed1f320ba1c176d97f37052a84353896b5d56cd8ed814af957fcbbed0d" exitCode=0 Jul 06 16:40:09.199939 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.199437 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1f505df-8e0b-41ce-9564-dccd86f560a5","Type":"ContainerDied","Data":"365d40ed1f320ba1c176d97f37052a84353896b5d56cd8ed814af957fcbbed0d"} Jul 06 16:40:09.251937 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.251835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-secret-metrics-server-tls\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.251937 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.251909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d7e15d77-2583-4602-b954-7ebee2216a26-audit-log\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.252155 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.251938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-secret-metrics-server-client-certs\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.252155 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.251980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-client-ca-bundle\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.252155 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.252016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6lc2\" (UniqueName: \"kubernetes.io/projected/d7e15d77-2583-4602-b954-7ebee2216a26-kube-api-access-b6lc2\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.252155 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.252050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d7e15d77-2583-4602-b954-7ebee2216a26-metrics-server-audit-profiles\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.252155 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.252076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e15d77-2583-4602-b954-7ebee2216a26-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.252481 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.252421 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d7e15d77-2583-4602-b954-7ebee2216a26-audit-log\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.252855 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.252830 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e15d77-2583-4602-b954-7ebee2216a26-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.253281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.253254 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d7e15d77-2583-4602-b954-7ebee2216a26-metrics-server-audit-profiles\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.254857 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.254832 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-client-ca-bundle\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.255053 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.255033 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-secret-metrics-server-tls\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.255276 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.255254 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d7e15d77-2583-4602-b954-7ebee2216a26-secret-metrics-server-client-certs\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.283300 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.283271 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6lc2\" (UniqueName: \"kubernetes.io/projected/d7e15d77-2583-4602-b954-7ebee2216a26-kube-api-access-b6lc2\") pod \"metrics-server-66b9d4f887-5xftb\" (UID: \"d7e15d77-2583-4602-b954-7ebee2216a26\") " pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.357889 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.357849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:09.633388 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:09.633315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66b9d4f887-5xftb"] Jul 06 16:40:09.673309 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:40:09.673278 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e15d77_2583_4602_b954_7ebee2216a26.slice/crio-5a129376223606bf675a0e5c90cf75e56714fbfb16a2f3e4932f469f02950960 WatchSource:0}: Error finding container 5a129376223606bf675a0e5c90cf75e56714fbfb16a2f3e4932f469f02950960: Status 404 returned error can't find the container with id 5a129376223606bf675a0e5c90cf75e56714fbfb16a2f3e4932f469f02950960 Jul 06 16:40:10.204112 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:10.204075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" event={"ID":"d7e15d77-2583-4602-b954-7ebee2216a26","Type":"ContainerStarted","Data":"5a129376223606bf675a0e5c90cf75e56714fbfb16a2f3e4932f469f02950960"} Jul 06 16:40:12.215611 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:12.215509 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1f505df-8e0b-41ce-9564-dccd86f560a5","Type":"ContainerStarted","Data":"8333f1db6fedd219b41976f98347cd97efd5d0de485cc56c91c4cd5b08a31de9"} Jul 06 16:40:12.216039 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:12.215630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1f505df-8e0b-41ce-9564-dccd86f560a5","Type":"ContainerStarted","Data":"5cddec8a6fc2a6d3573be8a2348fad7a17e7d625f2c8281dcbc2a489af73a047"} Jul 06 16:40:12.216039 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:12.215676 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1f505df-8e0b-41ce-9564-dccd86f560a5","Type":"ContainerStarted","Data":"81660a0bfdccf46e6f7572b4a7c62d91a06e8f1f853cb5047c64b6fc123c7318"} Jul 06 16:40:12.216039 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:12.215703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1f505df-8e0b-41ce-9564-dccd86f560a5","Type":"ContainerStarted","Data":"6412fa43a630f09a142366cf3f6f3941d5936f944b0f2b8721e88dd26392ad9e"} Jul 06 16:40:12.216039 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:12.215725 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1f505df-8e0b-41ce-9564-dccd86f560a5","Type":"ContainerStarted","Data":"34cb73388def6e3d6623982e9fad2df512e5e164a42a5c574627a3519f496260"} Jul 06 16:40:12.217091 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:12.217065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" event={"ID":"d7e15d77-2583-4602-b954-7ebee2216a26","Type":"ContainerStarted","Data":"efcf83798932d2d96cc7eb05ac93f5560c871b9fb41f703e4e1aaca3c57db3e7"} Jul 06 16:40:12.259181 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:12.259128 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" podStartSLOduration=2.26858781 podStartE2EDuration="4.259113065s" totalImagesPullingTime="1.990525255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:40:12.258841456 +0000 UTC m=+109.028733000" watchObservedRunningTime="2026-07-06 16:40:12.259113065 +0000 UTC m=+109.029004606" Jul 06 16:40:14.228351 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:14.228312 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1f505df-8e0b-41ce-9564-dccd86f560a5","Type":"ContainerStarted","Data":"b58f96692ea0008a31a0e74a93fca9f36f19f2342d3c8b8ad15553182accf751"} Jul 06 16:40:14.286873 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:14.286811 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.861289382 podStartE2EDuration="9.286794962s" totalImagesPullingTime="6.41313081s" totalInitContainerRuntime="12.37477ms" isStatefulPod=false podCreationTimestamp="2026-07-06 16:40:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:40:14.285870589 +0000 UTC m=+111.055762130" watchObservedRunningTime="2026-07-06 16:40:14.286794962 +0000 UTC m=+111.056686504" Jul 06 16:40:18.082746 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:18.082711 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:18.193234 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:18.193201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:18.991632 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:18.991583 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:19.155211 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:19.155170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:19.299554 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:19.299426 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:20.190236 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:20.190202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:20.358670 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:20.358634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:21.443639 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:21.443561 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:21.583750 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:21.583697 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:26.445662 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:26.445620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:26.831622 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:26.831585 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:27.254304 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:27.254254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:28.181571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.181509 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5d9b8ddff5-8bfqm" podUID="b3fced7a-1b73-4d4f-bb0b-23515519935e" containerName="console" containerID="cri-o://8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360" gracePeriod=15 Jul 06 16:40:28.433836 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.433770 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:40:28.505112 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-service-ca\") pod \"b3fced7a-1b73-4d4f-bb0b-23515519935e\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " Jul 06 16:40:28.505292 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505150 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-serving-cert\") pod \"b3fced7a-1b73-4d4f-bb0b-23515519935e\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " Jul 06 16:40:28.505292 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505178 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpl7q\" (UniqueName: \"kubernetes.io/projected/b3fced7a-1b73-4d4f-bb0b-23515519935e-kube-api-access-zpl7q\") pod \"b3fced7a-1b73-4d4f-bb0b-23515519935e\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " Jul 06 16:40:28.505292 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505221 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-config\") pod \"b3fced7a-1b73-4d4f-bb0b-23515519935e\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " Jul 06 16:40:28.505416 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505372 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-oauth-config\") pod \"b3fced7a-1b73-4d4f-bb0b-23515519935e\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " Jul 06 16:40:28.505513 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505497 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-oauth-serving-cert\") pod \"b3fced7a-1b73-4d4f-bb0b-23515519935e\" (UID: \"b3fced7a-1b73-4d4f-bb0b-23515519935e\") " Jul 06 16:40:28.505661 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505629 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-service-ca" (OuterVolumeSpecName: "service-ca") pod "b3fced7a-1b73-4d4f-bb0b-23515519935e" (UID: "b3fced7a-1b73-4d4f-bb0b-23515519935e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:40:28.505822 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505794 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-config" (OuterVolumeSpecName: "console-config") pod "b3fced7a-1b73-4d4f-bb0b-23515519935e" (UID: "b3fced7a-1b73-4d4f-bb0b-23515519935e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:40:28.505822 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505821 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:40:28.505991 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.505838 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b3fced7a-1b73-4d4f-bb0b-23515519935e" (UID: "b3fced7a-1b73-4d4f-bb0b-23515519935e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:40:28.507566 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.507544 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3fced7a-1b73-4d4f-bb0b-23515519935e-kube-api-access-zpl7q" (OuterVolumeSpecName: "kube-api-access-zpl7q") pod "b3fced7a-1b73-4d4f-bb0b-23515519935e" (UID: "b3fced7a-1b73-4d4f-bb0b-23515519935e"). InnerVolumeSpecName "kube-api-access-zpl7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:40:28.507830 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.507807 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b3fced7a-1b73-4d4f-bb0b-23515519935e" (UID: "b3fced7a-1b73-4d4f-bb0b-23515519935e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:40:28.507830 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.507821 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b3fced7a-1b73-4d4f-bb0b-23515519935e" (UID: "b3fced7a-1b73-4d4f-bb0b-23515519935e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:40:28.606816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.606765 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-serving-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:40:28.606816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.606807 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpl7q\" (UniqueName: \"kubernetes.io/projected/b3fced7a-1b73-4d4f-bb0b-23515519935e-kube-api-access-zpl7q\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:40:28.606816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.606818 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-config\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:40:28.606816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.606828 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3fced7a-1b73-4d4f-bb0b-23515519935e-console-oauth-config\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:40:28.606816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:28.606837 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3fced7a-1b73-4d4f-bb0b-23515519935e-oauth-serving-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:40:29.283631 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.283593 2574 generic.go:356] "Generic (PLEG): container finished" podID="b3fced7a-1b73-4d4f-bb0b-23515519935e" containerID="8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360" exitCode=0 Jul 06 16:40:29.284084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.283641 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9b8ddff5-8bfqm" Jul 06 16:40:29.284084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.283675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9b8ddff5-8bfqm" event={"ID":"b3fced7a-1b73-4d4f-bb0b-23515519935e","Type":"ContainerDied","Data":"8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360"} Jul 06 16:40:29.284084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.283726 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9b8ddff5-8bfqm" event={"ID":"b3fced7a-1b73-4d4f-bb0b-23515519935e","Type":"ContainerDied","Data":"a5e2dc4cdea0a21ebeb8d98e5e2e72f085424c2c78a108dad691ab6e22d07f54"} Jul 06 16:40:29.284084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.283746 2574 scope.go:117] "RemoveContainer" containerID="8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360" Jul 06 16:40:29.292200 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.292181 2574 scope.go:117] "RemoveContainer" containerID="8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360" Jul 06 16:40:29.292473 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:40:29.292432 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360\": container with ID starting with 8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360 not found: ID does not exist" containerID="8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360" Jul 06 16:40:29.292552 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.292474 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360"} err="failed to get container status \"8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360\": rpc error: code = NotFound desc = could not find container \"8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360\": container with ID starting with 8244715a775e078002f31942345ee2faff4d53cefe2f48656870bc9ce66ec360 not found: ID does not exist" Jul 06 16:40:29.313820 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.313792 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9b8ddff5-8bfqm"] Jul 06 16:40:29.319107 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.319084 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d9b8ddff5-8bfqm"] Jul 06 16:40:29.358129 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.358099 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:29.358308 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.358164 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:29.747401 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:29.747365 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3fced7a-1b73-4d4f-bb0b-23515519935e" path="/var/lib/kubelet/pods/b3fced7a-1b73-4d4f-bb0b-23515519935e/volumes" Jul 06 16:40:35.190369 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:35.190335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:35.314425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:35.314399 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:49.366948 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:49.366920 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:49.370903 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:49.370877 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66b9d4f887-5xftb" Jul 06 16:40:50.180816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:50.180774 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:40:50.311496 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:40:50.311443 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:41:26.718521 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:41:26.718484 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:41:26.908184 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:41:26.908142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:41:28.455241 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:41:28.455207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:42:00.021910 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:42:00.021878 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:42:11.312600 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:42:11.312565 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:43:23.615002 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:43:23.614975 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 16:45:09.210216 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:09.210183 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:10.596347 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:10.596312 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:12.212214 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:12.212187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:12.286549 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:12.286508 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:12.475801 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:12.475697 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:12.597444 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:12.597406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:14.285948 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.285916 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-546c965ccc-8nzkf"] Jul 06 16:45:14.286340 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.286221 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3fced7a-1b73-4d4f-bb0b-23515519935e" containerName="console" Jul 06 16:45:14.286340 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.286231 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3fced7a-1b73-4d4f-bb0b-23515519935e" containerName="console" Jul 06 16:45:14.286340 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.286274 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3fced7a-1b73-4d4f-bb0b-23515519935e" containerName="console" Jul 06 16:45:14.289178 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.289160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.291801 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.291773 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 16:45:14.292078 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.292054 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 16:45:14.292078 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.292064 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 16:45:14.292275 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.292088 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 16:45:14.292351 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.292335 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 16:45:14.292503 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.292486 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-584dk\"" Jul 06 16:45:14.297317 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.297284 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 16:45:14.318346 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.318313 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-546c965ccc-8nzkf"] Jul 06 16:45:14.443643 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.443600 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-telemeter-trusted-ca-bundle\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.443643 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.443644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-serving-certs-ca-bundle\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.443883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.443732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.443883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.443766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-telemeter-client-tls\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.443883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.443790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-metrics-client-ca\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.443883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.443808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxsr\" (UniqueName: \"kubernetes.io/projected/77755948-4df5-46cd-af5f-c99a51777479-kube-api-access-ldxsr\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.443883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.443847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-secret-telemeter-client\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.443883 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.443867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-federate-client-tls\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.544773 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.544672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.544773 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.544721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-telemeter-client-tls\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.545084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.544785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-metrics-client-ca\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.545084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.544808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxsr\" (UniqueName: \"kubernetes.io/projected/77755948-4df5-46cd-af5f-c99a51777479-kube-api-access-ldxsr\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.545084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.544834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-secret-telemeter-client\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.545084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.544855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-federate-client-tls\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.545084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.544911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-telemeter-trusted-ca-bundle\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.545084 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.544947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-serving-certs-ca-bundle\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.545659 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.545630 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-metrics-client-ca\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.545839 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.545660 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-serving-certs-ca-bundle\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.546118 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.546091 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77755948-4df5-46cd-af5f-c99a51777479-telemeter-trusted-ca-bundle\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.547330 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.547301 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.547499 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.547480 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-federate-client-tls\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.547837 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.547814 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-secret-telemeter-client\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.547896 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.547823 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77755948-4df5-46cd-af5f-c99a51777479-telemeter-client-tls\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.553807 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.553779 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxsr\" (UniqueName: \"kubernetes.io/projected/77755948-4df5-46cd-af5f-c99a51777479-kube-api-access-ldxsr\") pod \"telemeter-client-546c965ccc-8nzkf\" (UID: \"77755948-4df5-46cd-af5f-c99a51777479\") " pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.599474 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.599420 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" Jul 06 16:45:14.746990 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.746946 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-546c965ccc-8nzkf"] Jul 06 16:45:14.750426 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:45:14.750398 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77755948_4df5_46cd_af5f_c99a51777479.slice/crio-91dabf6a63a99c9124f2e6848c23ee093d2a96d9874a528d77c72358567341c8 WatchSource:0}: Error finding container 91dabf6a63a99c9124f2e6848c23ee093d2a96d9874a528d77c72358567341c8: Status 404 returned error can't find the container with id 91dabf6a63a99c9124f2e6848c23ee093d2a96d9874a528d77c72358567341c8 Jul 06 16:45:14.752953 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:14.752933 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:45:15.285709 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.285675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" event={"ID":"77755948-4df5-46cd-af5f-c99a51777479","Type":"ContainerStarted","Data":"91dabf6a63a99c9124f2e6848c23ee093d2a96d9874a528d77c72358567341c8"} Jul 06 16:45:15.499831 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.499787 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:15.502220 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.502197 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 16:45:15.505978 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.505953 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.508606 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.508576 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 16:45:15.508769 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.508749 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 16:45:15.509344 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.509311 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 16:45:15.509344 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.509330 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 16:45:15.509736 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.509712 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 16:45:15.509736 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.509722 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-atgauntn60e84\"" Jul 06 16:45:15.510091 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.510071 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 16:45:15.510209 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.510189 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 16:45:15.510291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.510122 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 16:45:15.510291 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.510230 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vwp24\"" Jul 06 16:45:15.510400 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.510163 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 16:45:15.510791 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.510772 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 16:45:15.510873 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.510792 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 16:45:15.511078 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.510981 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 16:45:15.513024 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.513006 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 16:45:15.515141 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.515121 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 16:45:15.524905 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.524877 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 16:45:15.552296 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552296 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552296 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-config\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552424 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-web-config\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dp5\" (UniqueName: \"kubernetes.io/projected/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-kube-api-access-k4dp5\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.552601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.553054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.553054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.553054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.553054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.553054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.553054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.553054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552822 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-config-out\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.553054 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.552843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.653937 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.653891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.653937 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.653933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.653937 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.653949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654260 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-web-config\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654260 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dp5\" (UniqueName: \"kubernetes.io/projected/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-kube-api-access-k4dp5\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654260 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654260 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654260 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654260 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-config-out\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654992 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654992 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-config\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654992 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654698 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654992 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654790 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.654992 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.654820 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.655243 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.655193 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.656110 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.655768 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.657149 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.657050 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.659131 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.659091 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.659240 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.659149 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.659334 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.659312 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.659398 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.659378 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-config-out\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.659478 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.659380 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.659542 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.659528 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.659670 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.659650 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-web-config\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.659750 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.659729 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.660057 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.660027 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.660293 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.660269 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.660498 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.660481 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.660566 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.660512 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.661893 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.661876 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-config\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.663800 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.663777 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dp5\" (UniqueName: \"kubernetes.io/projected/0eaeddba-2751-4fbd-8bde-6a3a90f6f07e-kube-api-access-k4dp5\") pod \"prometheus-k8s-0\" (UID: \"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.820150 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.820047 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:15.989972 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:15.989936 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 16:45:15.995049 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:45:15.995015 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eaeddba_2751_4fbd_8bde_6a3a90f6f07e.slice/crio-d468e0344246a683542ea352edf6bed924fcd620fc7272dee07efc583b17a85c WatchSource:0}: Error finding container d468e0344246a683542ea352edf6bed924fcd620fc7272dee07efc583b17a85c: Status 404 returned error can't find the container with id d468e0344246a683542ea352edf6bed924fcd620fc7272dee07efc583b17a85c Jul 06 16:45:16.292301 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:16.292058 2574 generic.go:356] "Generic (PLEG): container finished" podID="0eaeddba-2751-4fbd-8bde-6a3a90f6f07e" containerID="2908e2f50503810dd685f814349c0c751d559726464fc17bb579094dd2fe290b" exitCode=0 Jul 06 16:45:16.292301 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:16.292123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e","Type":"ContainerDied","Data":"2908e2f50503810dd685f814349c0c751d559726464fc17bb579094dd2fe290b"} Jul 06 16:45:16.292301 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:16.292252 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e","Type":"ContainerStarted","Data":"d468e0344246a683542ea352edf6bed924fcd620fc7272dee07efc583b17a85c"} Jul 06 16:45:17.298077 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:17.298044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" event={"ID":"77755948-4df5-46cd-af5f-c99a51777479","Type":"ContainerStarted","Data":"37e9d1f04994d4798a3c1af6d56520cfe4ea7248788b0be9afa7bf01d71276b6"} Jul 06 16:45:17.298510 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:17.298115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" event={"ID":"77755948-4df5-46cd-af5f-c99a51777479","Type":"ContainerStarted","Data":"7311d09d79c178121fc8966f22332a2c9e10db0c7943f09b8957cacf8686bc65"} Jul 06 16:45:17.298510 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:17.298135 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" event={"ID":"77755948-4df5-46cd-af5f-c99a51777479","Type":"ContainerStarted","Data":"3e70970378046a7d4045a4ab470c530976fd00040615d7f2ae7a31b9e89d287c"} Jul 06 16:45:17.342493 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:17.342424 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-546c965ccc-8nzkf" podStartSLOduration=1.240682189 podStartE2EDuration="3.342404568s" totalImagesPullingTime="2.101722379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:45:17.341074261 +0000 UTC m=+414.110965804" watchObservedRunningTime="2026-07-06 16:45:17.342404568 +0000 UTC m=+414.112296134" Jul 06 16:45:18.253614 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:18.253572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:18.331973 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:18.331938 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:21.318192 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:21.318151 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e","Type":"ContainerStarted","Data":"b4a10a93d176a47f74f1c1178ed2177d61ed58eef617a77fa12b9e3b7a57071e"} Jul 06 16:45:21.318701 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:21.318260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e","Type":"ContainerStarted","Data":"9e32cd9e1b5a744e363313686e17607948a2101dba6eb24f55630e1b109ffa43"} Jul 06 16:45:23.329127 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:23.329033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e","Type":"ContainerStarted","Data":"3b90764ec2e8da03cf464c029307cbbcb57c7d314138cad7a33686106212fdba"} Jul 06 16:45:23.329619 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:23.329147 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e","Type":"ContainerStarted","Data":"787e43b339c2fb0e5f1546c48495bfd2bea0e92d1f07d122ce06e079f8c342a8"} Jul 06 16:45:23.329619 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:23.329192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e","Type":"ContainerStarted","Data":"14705672b8b97d7fbfc177d84376ed58627bda359c248ec86f15612703723ecb"} Jul 06 16:45:23.329619 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:23.329230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eaeddba-2751-4fbd-8bde-6a3a90f6f07e","Type":"ContainerStarted","Data":"c0d6a7fd1ba2add3c91ead9967fcbcc205fd02f969a50049cc16102c00007eb5"} Jul 06 16:45:23.365406 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:23.365352 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.747349525 podStartE2EDuration="8.365334235s" totalImagesPullingTime="6.599020855s" totalInitContainerRuntime="18.963855ms" isStatefulPod=false podCreationTimestamp="2026-07-06 16:45:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:45:23.364110502 +0000 UTC m=+420.134002077" watchObservedRunningTime="2026-07-06 16:45:23.365334235 +0000 UTC m=+420.135225776" Jul 06 16:45:25.820552 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:25.820518 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:45:28.796298 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:28.796265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:28.898941 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:28.898908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:45:28.981264 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:45:28.981237 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:12.621875 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:12.621823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:15.820972 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:15.820933 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:46:15.841034 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:15.841006 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:46:16.550478 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:16.550438 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 16:46:25.392115 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:25.392085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:25.462672 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:25.462634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:26.873398 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:26.873361 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:26.953357 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:26.953329 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:37.327926 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:37.327891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:37.396627 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:37.396591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:38.020741 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:38.020701 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:38.121481 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:38.121427 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:38.203405 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:38.203378 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:50.147982 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:50.147949 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:50.244356 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:50.244326 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:46:50.311716 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:46:50.311686 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:47:41.479283 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:47:41.479252 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:47:41.579775 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:47:41.579745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:47:42.895788 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:47:42.895759 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:47:42.999446 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:47:42.999413 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:01.442050 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:01.442017 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:01.540075 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:01.540044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:10.029842 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:10.029804 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:10.942083 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:10.942049 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:14.271357 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:14.271319 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:14.381561 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:14.381526 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:16.247506 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:16.247472 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:16.416437 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:16.416407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:17.061187 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:17.061156 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:17.318710 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:17.318633 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:19.384104 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:19.384070 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:19.746763 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:19.746733 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:21.072021 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:21.071988 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:21.173742 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:21.173713 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:26.913710 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:26.913670 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:27.919798 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:27.919765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:28.044519 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:28.044489 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:28.528706 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:28.528680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:28.632817 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:28.632777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:32.544599 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:32.544567 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:32.651816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:32.651786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:34.479179 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:34.479137 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:47.048111 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:47.048039 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:47.132045 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:47.132005 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:50.623793 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:50.623753 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:50.723797 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:50.723762 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:51.267611 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.267580 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:51.568860 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.568757 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:51.666575 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.666548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:51.730024 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.729994 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz"] Jul 06 16:48:51.733380 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.733355 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.737799 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.737776 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 16:48:51.739703 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.739682 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 16:48:51.739980 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.739956 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kwj6w\"" Jul 06 16:48:51.740327 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.740308 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 16:48:51.740811 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.740327 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 16:48:51.756037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.756011 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz"] Jul 06 16:48:51.826971 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.826877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.826971 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.826918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.826971 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.826951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-manifests\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.827199 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.826983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vv45\" (UniqueName: \"kubernetes.io/projected/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-kube-api-access-6vv45\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.827199 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.827064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-tmp\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.927583 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.927542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-tmp\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.927789 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.927616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.927789 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.927648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.927789 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.927690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-manifests\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.927789 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.927722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vv45\" (UniqueName: \"kubernetes.io/projected/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-kube-api-access-6vv45\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.928109 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.928088 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-manifests\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.930127 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.930100 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-tmp\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.930221 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.930205 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.930260 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.930228 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.938093 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.938066 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vv45\" (UniqueName: \"kubernetes.io/projected/c0ea03ea-d2c5-46db-86e7-48012d5a16b7-kube-api-access-6vv45\") pod \"opendatahub-operator-controller-manager-d66bcd77d-ffxdz\" (UID: \"c0ea03ea-d2c5-46db-86e7-48012d5a16b7\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:51.944996 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:51.944937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:52.044633 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:52.044601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:52.045517 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:52.045493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:52.167593 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:52.167564 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:52.202215 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:52.202178 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz"] Jul 06 16:48:52.206446 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:48:52.206423 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ea03ea_d2c5_46db_86e7_48012d5a16b7.slice/crio-c89333d495c6e32eb0668b5ee11475f12b7bd77b8e232bd6b18fa6f00f5ccf84 WatchSource:0}: Error finding container c89333d495c6e32eb0668b5ee11475f12b7bd77b8e232bd6b18fa6f00f5ccf84: Status 404 returned error can't find the container with id c89333d495c6e32eb0668b5ee11475f12b7bd77b8e232bd6b18fa6f00f5ccf84 Jul 06 16:48:52.251886 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:52.251852 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:52.355688 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:52.355602 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:52.464057 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:52.464026 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:53.142582 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:53.142543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" event={"ID":"c0ea03ea-d2c5-46db-86e7-48012d5a16b7","Type":"ContainerStarted","Data":"c89333d495c6e32eb0668b5ee11475f12b7bd77b8e232bd6b18fa6f00f5ccf84"} Jul 06 16:48:56.160582 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:56.160549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" event={"ID":"c0ea03ea-d2c5-46db-86e7-48012d5a16b7","Type":"ContainerStarted","Data":"e09ace976e7046f7c0751512b2164fffc71ad87919533fd0eff56c46334be19a"} Jul 06 16:48:57.166517 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:57.166408 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0ea03ea-d2c5-46db-86e7-48012d5a16b7" containerID="e09ace976e7046f7c0751512b2164fffc71ad87919533fd0eff56c46334be19a" exitCode=0 Jul 06 16:48:57.166517 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:57.166490 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" event={"ID":"c0ea03ea-d2c5-46db-86e7-48012d5a16b7","Type":"ContainerDied","Data":"e09ace976e7046f7c0751512b2164fffc71ad87919533fd0eff56c46334be19a"} Jul 06 16:48:58.173050 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:58.173013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" event={"ID":"c0ea03ea-d2c5-46db-86e7-48012d5a16b7","Type":"ContainerStarted","Data":"71f7269002221b3263e3bee25fb9413ba8c3fd425b4f22205d684133322ae556"} Jul 06 16:48:58.173553 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:58.173142 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:48:58.216268 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:58.216211 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" podStartSLOduration=2.292295761 podStartE2EDuration="7.216192868s" totalImagesPullingTime="3.879557402s" totalInitContainerRuntime="1.044339705s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:48:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:48:58.214803207 +0000 UTC m=+634.984694748" watchObservedRunningTime="2026-07-06 16:48:58.216192868 +0000 UTC m=+634.986084409" Jul 06 16:48:58.693925 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:58.693894 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:48:58.800835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:48:58.800786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:03.273629 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:03.273583 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:03.374168 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:03.374130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:09.180404 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:09.180372 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-ffxdz" Jul 06 16:49:10.945056 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:10.945022 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:11.049044 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:11.048998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:11.218427 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:11.218323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:11.399878 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:11.399848 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:12.143139 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:12.143104 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:12.330714 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:12.330684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:12.546903 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:12.546870 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:12.819966 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:12.819875 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:13.563142 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:13.563110 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:13.774433 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:13.774401 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:13.927227 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:13.927137 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:14.107278 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:14.107246 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:15.642601 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:15.642570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:18.585338 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:18.585305 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:18.694757 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:18.694725 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:19.536015 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:19.535978 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:19.879669 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:19.879570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:20.107626 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:20.107595 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:21.194853 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.194817 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb"] Jul 06 16:49:21.197943 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.197923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.202154 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.202132 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 16:49:21.204520 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.204295 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 16:49:21.204691 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.204668 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 16:49:21.204691 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.204696 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 16:49:21.204816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.204697 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g4s4q\"" Jul 06 16:49:21.204816 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.204737 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 16:49:21.213235 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.213201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb"] Jul 06 16:49:21.286616 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.286576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnwc\" (UniqueName: \"kubernetes.io/projected/cda095e1-cac1-4fdb-bad4-71643064b7e8-kube-api-access-cxnwc\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.286801 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.286627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda095e1-cac1-4fdb-bad4-71643064b7e8-metrics-cert\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.286801 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.286650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cda095e1-cac1-4fdb-bad4-71643064b7e8-manager-config\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.286801 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.286726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cda095e1-cac1-4fdb-bad4-71643064b7e8-cert\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.387407 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.387365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnwc\" (UniqueName: \"kubernetes.io/projected/cda095e1-cac1-4fdb-bad4-71643064b7e8-kube-api-access-cxnwc\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.387612 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.387416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda095e1-cac1-4fdb-bad4-71643064b7e8-metrics-cert\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.387612 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.387444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cda095e1-cac1-4fdb-bad4-71643064b7e8-manager-config\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.387612 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.387491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cda095e1-cac1-4fdb-bad4-71643064b7e8-cert\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.388067 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.388045 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cda095e1-cac1-4fdb-bad4-71643064b7e8-manager-config\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.390082 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.390049 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cda095e1-cac1-4fdb-bad4-71643064b7e8-cert\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.390082 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.390080 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda095e1-cac1-4fdb-bad4-71643064b7e8-metrics-cert\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.420703 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.420662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnwc\" (UniqueName: \"kubernetes.io/projected/cda095e1-cac1-4fdb-bad4-71643064b7e8-kube-api-access-cxnwc\") pod \"lws-controller-manager-5c47b9fd86-hz9lb\" (UID: \"cda095e1-cac1-4fdb-bad4-71643064b7e8\") " pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.507645 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.507611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:21.654815 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:21.654783 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb"] Jul 06 16:49:21.661358 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:49:21.661327 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda095e1_cac1_4fdb_bad4_71643064b7e8.slice/crio-22b0c213163f170f99671ba0e1d95c44f1dd63f6293838dd922a37b9fcf99186 WatchSource:0}: Error finding container 22b0c213163f170f99671ba0e1d95c44f1dd63f6293838dd922a37b9fcf99186: Status 404 returned error can't find the container with id 22b0c213163f170f99671ba0e1d95c44f1dd63f6293838dd922a37b9fcf99186 Jul 06 16:49:22.274810 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.274767 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" event={"ID":"cda095e1-cac1-4fdb-bad4-71643064b7e8","Type":"ContainerStarted","Data":"22b0c213163f170f99671ba0e1d95c44f1dd63f6293838dd922a37b9fcf99186"} Jul 06 16:49:22.482285 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.482064 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9"] Jul 06 16:49:22.485757 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.485720 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.490676 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.490649 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 16:49:22.491035 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.491012 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 16:49:22.491035 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.491023 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 16:49:22.491758 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.491374 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wcv2l\"" Jul 06 16:49:22.491758 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.491707 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 16:49:22.492254 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.492049 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 16:49:22.492254 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.492066 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 16:49:22.507345 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.507286 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9"] Jul 06 16:49:22.597251 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.597216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfxs\" (UniqueName: \"kubernetes.io/projected/74bd927a-4a82-47c4-b177-860aa330b4dd-kube-api-access-csfxs\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.597474 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.597280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.597474 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.597313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.597474 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.597381 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.597474 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.597410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bd927a-4a82-47c4-b177-860aa330b4dd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.597474 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.597468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.597750 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.597492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.698135 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.698095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfxs\" (UniqueName: \"kubernetes.io/projected/74bd927a-4a82-47c4-b177-860aa330b4dd-kube-api-access-csfxs\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.698294 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.698168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.698294 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.698197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.698294 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.698248 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.698294 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.698271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bd927a-4a82-47c4-b177-860aa330b4dd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.698444 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.698319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.698444 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.698343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.699419 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.699390 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.701253 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.701231 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.701438 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.701414 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.701516 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.701489 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.701598 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.701583 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bd927a-4a82-47c4-b177-860aa330b4dd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.726504 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.726472 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bd927a-4a82-47c4-b177-860aa330b4dd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.727068 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.727047 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfxs\" (UniqueName: \"kubernetes.io/projected/74bd927a-4a82-47c4-b177-860aa330b4dd-kube-api-access-csfxs\") pod \"istiod-openshift-gateway-94bb8fbfd-hqzt9\" (UID: \"74bd927a-4a82-47c4-b177-860aa330b4dd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.800534 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.800429 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:22.976314 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:22.976285 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9"] Jul 06 16:49:23.797159 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:23.797113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:23.871606 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:49:23.871571 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bd927a_4a82_47c4_b177_860aa330b4dd.slice/crio-a4d1a6169f0eb998a9b32a683ff5f030cd69afa49cf28bd9478aaf83bfe8430e WatchSource:0}: Error finding container a4d1a6169f0eb998a9b32a683ff5f030cd69afa49cf28bd9478aaf83bfe8430e: Status 404 returned error can't find the container with id a4d1a6169f0eb998a9b32a683ff5f030cd69afa49cf28bd9478aaf83bfe8430e Jul 06 16:49:23.924142 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:23.924109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:24.151118 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:24.151039 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:24.250075 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:24.250034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:24.286397 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:24.286355 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" event={"ID":"cda095e1-cac1-4fdb-bad4-71643064b7e8","Type":"ContainerStarted","Data":"2c3cbb7ef8581db4cd58388c80d369ccd4d807a52e2e5454b5f9f1d3657e4b6e"} Jul 06 16:49:24.286607 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:24.286491 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:49:24.287626 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:24.287602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" event={"ID":"74bd927a-4a82-47c4-b177-860aa330b4dd","Type":"ContainerStarted","Data":"a4d1a6169f0eb998a9b32a683ff5f030cd69afa49cf28bd9478aaf83bfe8430e"} Jul 06 16:49:24.318607 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:24.318534 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" podStartSLOduration=1.048984098 podStartE2EDuration="3.318511125s" totalImagesPullingTime="2.269527027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:49:24.318223892 +0000 UTC m=+661.088115434" watchObservedRunningTime="2026-07-06 16:49:24.318511125 +0000 UTC m=+661.088402668" Jul 06 16:49:24.353241 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:24.353195 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:49:26.035635 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:26.035598 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 16:49:26.036005 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:26.035676 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 16:49:26.299440 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:26.299345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" event={"ID":"74bd927a-4a82-47c4-b177-860aa330b4dd","Type":"ContainerStarted","Data":"8569975b4c8373bba030767c998e7a68aabbc0a72bdd7fcebacbc69ae9133039"} Jul 06 16:49:26.299636 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:26.299488 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:26.324280 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:26.324220 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" podStartSLOduration=2.162961506 podStartE2EDuration="4.324195052s" totalImagesPullingTime="2.161233546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:49:26.32277634 +0000 UTC m=+663.092667880" watchObservedRunningTime="2026-07-06 16:49:26.324195052 +0000 UTC m=+663.094086597" Jul 06 16:49:27.305610 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:27.305582 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hqzt9" Jul 06 16:49:35.294337 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:49:35.294307 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c47b9fd86-hz9lb" Jul 06 16:50:12.547920 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:12.547884 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:13.757964 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:13.757930 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:19.691693 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:19.691611 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:19.757031 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:19.756996 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:21.400047 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:21.400011 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:21.468604 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:21.468573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:31.873729 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:31.873690 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:31.974749 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:31.974712 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:32.048739 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:32.048706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:41.106906 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:41.106871 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:44.539501 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:44.539444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:46.510309 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:46.510276 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:48.114907 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:48.114863 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:48.494976 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:48.494942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:49.939378 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:49.939342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:51.504066 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:51.504014 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:52.336854 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:52.336816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:54.335207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:54.335177 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:56.150345 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:56.150307 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:56.249038 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:56.249005 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:56.550720 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:56.550684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:50:56.706155 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:50:56.706116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:03.735974 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:03.735940 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:03.857218 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:03.857180 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:07.685069 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.685038 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd"] Jul 06 16:51:07.688245 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.688222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" Jul 06 16:51:07.691686 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.691662 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mzdrb\"" Jul 06 16:51:07.691834 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.691664 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 16:51:07.692384 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.692362 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 16:51:07.704842 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.704809 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd"] Jul 06 16:51:07.860268 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.860225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2v4s\" (UniqueName: \"kubernetes.io/projected/e764667a-8a18-4668-8e93-323e6383a810-kube-api-access-c2v4s\") pod \"limitador-operator-controller-manager-859d7b78bd-c9nxd\" (UID: \"e764667a-8a18-4668-8e93-323e6383a810\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" Jul 06 16:51:07.961729 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.961627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2v4s\" (UniqueName: \"kubernetes.io/projected/e764667a-8a18-4668-8e93-323e6383a810-kube-api-access-c2v4s\") pod \"limitador-operator-controller-manager-859d7b78bd-c9nxd\" (UID: \"e764667a-8a18-4668-8e93-323e6383a810\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" Jul 06 16:51:07.986253 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:07.986220 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2v4s\" (UniqueName: \"kubernetes.io/projected/e764667a-8a18-4668-8e93-323e6383a810-kube-api-access-c2v4s\") pod \"limitador-operator-controller-manager-859d7b78bd-c9nxd\" (UID: \"e764667a-8a18-4668-8e93-323e6383a810\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" Jul 06 16:51:08.000066 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:08.000033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" Jul 06 16:51:08.139838 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:08.139805 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd"] Jul 06 16:51:08.143491 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:51:08.143443 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode764667a_8a18_4668_8e93_323e6383a810.slice/crio-4930e2e9512e7d8e41e40c13e7ecb377e2de1b9905544c8ee15351e35b6b525e WatchSource:0}: Error finding container 4930e2e9512e7d8e41e40c13e7ecb377e2de1b9905544c8ee15351e35b6b525e: Status 404 returned error can't find the container with id 4930e2e9512e7d8e41e40c13e7ecb377e2de1b9905544c8ee15351e35b6b525e Jul 06 16:51:08.145372 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:08.145355 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:51:08.535165 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:08.535133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:08.638225 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:08.638179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:08.735871 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:08.735840 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" event={"ID":"e764667a-8a18-4668-8e93-323e6383a810","Type":"ContainerStarted","Data":"4930e2e9512e7d8e41e40c13e7ecb377e2de1b9905544c8ee15351e35b6b525e"} Jul 06 16:51:10.746904 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:10.746867 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" event={"ID":"e764667a-8a18-4668-8e93-323e6383a810","Type":"ContainerStarted","Data":"5ad278f36fdc2edc4436beed46612c0716b75ec0bfac98ef711f5156b051a279"} Jul 06 16:51:10.747410 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:10.747001 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" Jul 06 16:51:10.771469 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:10.766892 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" podStartSLOduration=1.88716745 podStartE2EDuration="3.766870392s" totalImagesPullingTime="1.879702942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:51:10.764498333 +0000 UTC m=+767.534389875" watchObservedRunningTime="2026-07-06 16:51:10.766870392 +0000 UTC m=+767.536761939" Jul 06 16:51:10.994103 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:10.994068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:11.094950 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:11.094867 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:16.385205 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:16.385170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:16.485227 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:16.485193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:16.585946 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:16.585912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:17.195047 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:17.195015 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:21.753586 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:21.753560 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c9nxd" Jul 06 16:51:23.463610 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:23.463578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:23.564036 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:23.564004 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:28.258996 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:28.258957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:28.359152 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:28.359119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:30.749382 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:30.749352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:30.850511 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:30.850482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:36.464850 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:36.464813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:36.530118 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:36.530084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:38.710894 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:38.710854 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:38.766037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:38.766011 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:43.884964 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:43.884931 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:43.982093 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:43.982059 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:48.685874 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:48.685800 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:48.786055 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:48.786022 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:50.371351 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:50.371315 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:50.663970 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:50.663878 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:51.146928 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:51.146893 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:51.215413 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:51.215380 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:56.318180 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:56.318144 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:56.468248 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:56.468216 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:56.617256 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:56.617179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:57.392610 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:57.392575 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:57.499077 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:57.499042 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:51:59.057143 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:51:59.057111 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:17.821322 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:17.821282 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:30.623918 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:30.623876 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:31.134779 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:31.134743 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:37.470751 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.470715 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:37.478734 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.478700 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ff8768844-wksfh"] Jul 06 16:52:37.482066 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.482042 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:37.484595 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.484565 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 16:52:37.484711 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.484612 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 16:52:37.484711 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.484618 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b857d\"" Jul 06 16:52:37.494832 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.494730 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff8768844-wksfh"] Jul 06 16:52:37.570234 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.570203 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:37.575320 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.575284 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2760a17-0d84-404f-b0d6-e593d91aa137-webhook-cert\") pod \"maas-controller-7ff8768844-wksfh\" (UID: \"b2760a17-0d84-404f-b0d6-e593d91aa137\") " pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:37.575511 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.575340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72mm\" (UniqueName: \"kubernetes.io/projected/b2760a17-0d84-404f-b0d6-e593d91aa137-kube-api-access-v72mm\") pod \"maas-controller-7ff8768844-wksfh\" (UID: \"b2760a17-0d84-404f-b0d6-e593d91aa137\") " pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:37.657069 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.657034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:37.663174 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.663144 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-589c49595b-82csr"] Jul 06 16:52:37.666339 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.666318 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:37.676109 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.676076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2760a17-0d84-404f-b0d6-e593d91aa137-webhook-cert\") pod \"maas-controller-7ff8768844-wksfh\" (UID: \"b2760a17-0d84-404f-b0d6-e593d91aa137\") " pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:37.676268 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.676139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72mm\" (UniqueName: \"kubernetes.io/projected/b2760a17-0d84-404f-b0d6-e593d91aa137-kube-api-access-v72mm\") pod \"maas-controller-7ff8768844-wksfh\" (UID: \"b2760a17-0d84-404f-b0d6-e593d91aa137\") " pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:37.677811 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.677788 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-589c49595b-82csr"] Jul 06 16:52:37.679063 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.679043 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2760a17-0d84-404f-b0d6-e593d91aa137-webhook-cert\") pod \"maas-controller-7ff8768844-wksfh\" (UID: \"b2760a17-0d84-404f-b0d6-e593d91aa137\") " pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:37.684536 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.684510 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72mm\" (UniqueName: \"kubernetes.io/projected/b2760a17-0d84-404f-b0d6-e593d91aa137-kube-api-access-v72mm\") pod \"maas-controller-7ff8768844-wksfh\" (UID: \"b2760a17-0d84-404f-b0d6-e593d91aa137\") " pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:37.777607 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.777571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn979\" (UniqueName: \"kubernetes.io/projected/8f787297-0ba3-417a-90cc-d1866ebe0874-kube-api-access-tn979\") pod \"maas-controller-589c49595b-82csr\" (UID: \"8f787297-0ba3-417a-90cc-d1866ebe0874\") " pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:37.777835 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.777621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f787297-0ba3-417a-90cc-d1866ebe0874-webhook-cert\") pod \"maas-controller-589c49595b-82csr\" (UID: \"8f787297-0ba3-417a-90cc-d1866ebe0874\") " pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:37.791649 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.791619 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:37.879039 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.879007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f787297-0ba3-417a-90cc-d1866ebe0874-webhook-cert\") pod \"maas-controller-589c49595b-82csr\" (UID: \"8f787297-0ba3-417a-90cc-d1866ebe0874\") " pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:37.879212 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.879113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn979\" (UniqueName: \"kubernetes.io/projected/8f787297-0ba3-417a-90cc-d1866ebe0874-kube-api-access-tn979\") pod \"maas-controller-589c49595b-82csr\" (UID: \"8f787297-0ba3-417a-90cc-d1866ebe0874\") " pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:37.881706 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.881673 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f787297-0ba3-417a-90cc-d1866ebe0874-webhook-cert\") pod \"maas-controller-589c49595b-82csr\" (UID: \"8f787297-0ba3-417a-90cc-d1866ebe0874\") " pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:37.888226 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.888193 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn979\" (UniqueName: \"kubernetes.io/projected/8f787297-0ba3-417a-90cc-d1866ebe0874-kube-api-access-tn979\") pod \"maas-controller-589c49595b-82csr\" (UID: \"8f787297-0ba3-417a-90cc-d1866ebe0874\") " pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:37.927890 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.927805 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff8768844-wksfh"] Jul 06 16:52:37.930752 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:52:37.930724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2760a17_0d84_404f_b0d6_e593d91aa137.slice/crio-526ae703bc53667adbda56086bb6c33acfc0f0988fe11d5ab714dfcc9daa8222 WatchSource:0}: Error finding container 526ae703bc53667adbda56086bb6c33acfc0f0988fe11d5ab714dfcc9daa8222: Status 404 returned error can't find the container with id 526ae703bc53667adbda56086bb6c33acfc0f0988fe11d5ab714dfcc9daa8222 Jul 06 16:52:37.977378 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:37.977340 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:38.114067 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:38.114041 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-589c49595b-82csr"] Jul 06 16:52:38.116902 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:52:38.116871 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f787297_0ba3_417a_90cc_d1866ebe0874.slice/crio-953583bfb2473b00b62a9bf44fdee16ed833833956782621c2ac76aaf1cc4ae1 WatchSource:0}: Error finding container 953583bfb2473b00b62a9bf44fdee16ed833833956782621c2ac76aaf1cc4ae1: Status 404 returned error can't find the container with id 953583bfb2473b00b62a9bf44fdee16ed833833956782621c2ac76aaf1cc4ae1 Jul 06 16:52:38.140630 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:38.140585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff8768844-wksfh" event={"ID":"b2760a17-0d84-404f-b0d6-e593d91aa137","Type":"ContainerStarted","Data":"526ae703bc53667adbda56086bb6c33acfc0f0988fe11d5ab714dfcc9daa8222"} Jul 06 16:52:38.141683 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:38.141663 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-589c49595b-82csr" event={"ID":"8f787297-0ba3-417a-90cc-d1866ebe0874","Type":"ContainerStarted","Data":"953583bfb2473b00b62a9bf44fdee16ed833833956782621c2ac76aaf1cc4ae1"} Jul 06 16:52:40.929848 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:40.929813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:41.027526 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:41.027492 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:41.158981 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:41.158948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff8768844-wksfh" event={"ID":"b2760a17-0d84-404f-b0d6-e593d91aa137","Type":"ContainerStarted","Data":"5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada"} Jul 06 16:52:41.159185 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:41.159070 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:41.160348 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:41.160327 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-589c49595b-82csr" event={"ID":"8f787297-0ba3-417a-90cc-d1866ebe0874","Type":"ContainerStarted","Data":"40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f"} Jul 06 16:52:41.160483 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:41.160449 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:41.183638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:41.183585 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ff8768844-wksfh" podStartSLOduration=1.380107485 podStartE2EDuration="4.18357097s" totalImagesPullingTime="2.803463485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:52:41.181369254 +0000 UTC m=+857.951260795" watchObservedRunningTime="2026-07-06 16:52:41.18357097 +0000 UTC m=+857.953462513" Jul 06 16:52:41.202295 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:41.202242 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-589c49595b-82csr" podStartSLOduration=1.574822717 podStartE2EDuration="4.202223676s" totalImagesPullingTime="2.627400959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:52:41.199766241 +0000 UTC m=+857.969657784" watchObservedRunningTime="2026-07-06 16:52:41.202223676 +0000 UTC m=+857.972115217" Jul 06 16:52:51.902245 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:51.902213 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:51.973180 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:51.973150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:52:52.172037 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.171940 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:52:52.172259 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.172241 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:52.289904 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.289867 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff8768844-wksfh"] Jul 06 16:52:52.290142 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.290119 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7ff8768844-wksfh" podUID="b2760a17-0d84-404f-b0d6-e593d91aa137" containerName="manager" containerID="cri-o://5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada" gracePeriod=30 Jul 06 16:52:52.546279 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.546255 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:52.612526 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.612492 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v72mm\" (UniqueName: \"kubernetes.io/projected/b2760a17-0d84-404f-b0d6-e593d91aa137-kube-api-access-v72mm\") pod \"b2760a17-0d84-404f-b0d6-e593d91aa137\" (UID: \"b2760a17-0d84-404f-b0d6-e593d91aa137\") " Jul 06 16:52:52.612727 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.612572 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2760a17-0d84-404f-b0d6-e593d91aa137-webhook-cert\") pod \"b2760a17-0d84-404f-b0d6-e593d91aa137\" (UID: \"b2760a17-0d84-404f-b0d6-e593d91aa137\") " Jul 06 16:52:52.614847 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.614820 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2760a17-0d84-404f-b0d6-e593d91aa137-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b2760a17-0d84-404f-b0d6-e593d91aa137" (UID: "b2760a17-0d84-404f-b0d6-e593d91aa137"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:52:52.615014 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.614774 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2760a17-0d84-404f-b0d6-e593d91aa137-kube-api-access-v72mm" (OuterVolumeSpecName: "kube-api-access-v72mm") pod "b2760a17-0d84-404f-b0d6-e593d91aa137" (UID: "b2760a17-0d84-404f-b0d6-e593d91aa137"). InnerVolumeSpecName "kube-api-access-v72mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:52:52.713592 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.713489 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72mm\" (UniqueName: \"kubernetes.io/projected/b2760a17-0d84-404f-b0d6-e593d91aa137-kube-api-access-v72mm\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:52:52.713592 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:52.713533 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2760a17-0d84-404f-b0d6-e593d91aa137-webhook-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:52:53.213953 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.213914 2574 generic.go:356] "Generic (PLEG): container finished" podID="b2760a17-0d84-404f-b0d6-e593d91aa137" containerID="5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada" exitCode=0 Jul 06 16:52:53.214382 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.213964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff8768844-wksfh" event={"ID":"b2760a17-0d84-404f-b0d6-e593d91aa137","Type":"ContainerDied","Data":"5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada"} Jul 06 16:52:53.214382 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.214030 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff8768844-wksfh" event={"ID":"b2760a17-0d84-404f-b0d6-e593d91aa137","Type":"ContainerDied","Data":"526ae703bc53667adbda56086bb6c33acfc0f0988fe11d5ab714dfcc9daa8222"} Jul 06 16:52:53.214382 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.214052 2574 scope.go:117] "RemoveContainer" containerID="5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada" Jul 06 16:52:53.214382 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.213977 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff8768844-wksfh" Jul 06 16:52:53.222763 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.222740 2574 scope.go:117] "RemoveContainer" containerID="5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada" Jul 06 16:52:53.223064 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:52:53.223045 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada\": container with ID starting with 5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada not found: ID does not exist" containerID="5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada" Jul 06 16:52:53.223116 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.223074 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada"} err="failed to get container status \"5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada\": rpc error: code = NotFound desc = could not find container \"5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada\": container with ID starting with 5cf7d925059d4682ff8fd39e241b2601680e105ae44644277cb715b9476eeada not found: ID does not exist" Jul 06 16:52:53.235871 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.235832 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff8768844-wksfh"] Jul 06 16:52:53.240722 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.240693 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ff8768844-wksfh"] Jul 06 16:52:53.747894 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:52:53.747859 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2760a17-0d84-404f-b0d6-e593d91aa137" path="/var/lib/kubelet/pods/b2760a17-0d84-404f-b0d6-e593d91aa137/volumes" Jul 06 16:53:13.281746 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.281707 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:13.816274 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.816238 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:13.877845 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.877817 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-78ddc7f974-4rtm2"] Jul 06 16:53:13.878304 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.878282 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2760a17-0d84-404f-b0d6-e593d91aa137" containerName="manager" Jul 06 16:53:13.878304 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.878304 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2760a17-0d84-404f-b0d6-e593d91aa137" containerName="manager" Jul 06 16:53:13.878519 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.878383 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2760a17-0d84-404f-b0d6-e593d91aa137" containerName="manager" Jul 06 16:53:13.884838 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.884808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:13.887286 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.887257 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 16:53:13.888028 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.888006 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jw2ph\"" Jul 06 16:53:13.893582 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.893553 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-78ddc7f974-4rtm2"] Jul 06 16:53:13.910836 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:13.910798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:14.015366 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.015321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c508e2-5ff7-46aa-9e63-c52efcb1128d-maas-api-tls\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.015366 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.015367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c508e2-5ff7-46aa-9e63-c52efcb1128d-openshift-service-ca\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.015621 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.015387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqlc\" (UniqueName: \"kubernetes.io/projected/03c508e2-5ff7-46aa-9e63-c52efcb1128d-kube-api-access-4wqlc\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.116730 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.116629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c508e2-5ff7-46aa-9e63-c52efcb1128d-maas-api-tls\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.116730 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.116670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c508e2-5ff7-46aa-9e63-c52efcb1128d-openshift-service-ca\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.116730 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.116689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqlc\" (UniqueName: \"kubernetes.io/projected/03c508e2-5ff7-46aa-9e63-c52efcb1128d-kube-api-access-4wqlc\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.117471 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.117434 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c508e2-5ff7-46aa-9e63-c52efcb1128d-openshift-service-ca\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.119278 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.119249 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c508e2-5ff7-46aa-9e63-c52efcb1128d-maas-api-tls\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.125781 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.125752 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqlc\" (UniqueName: \"kubernetes.io/projected/03c508e2-5ff7-46aa-9e63-c52efcb1128d-kube-api-access-4wqlc\") pod \"maas-api-78ddc7f974-4rtm2\" (UID: \"03c508e2-5ff7-46aa-9e63-c52efcb1128d\") " pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.220983 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.220949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:14.264666 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.264629 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:14.364094 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.364060 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:14.393431 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.393402 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-78ddc7f974-4rtm2"] Jul 06 16:53:14.396045 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:53:14.396014 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c508e2_5ff7_46aa_9e63_c52efcb1128d.slice/crio-e2e08e3b34b0f086fbd77e6a610923d47363f8932b093c5d83d60da65f1b3f90 WatchSource:0}: Error finding container e2e08e3b34b0f086fbd77e6a610923d47363f8932b093c5d83d60da65f1b3f90: Status 404 returned error can't find the container with id e2e08e3b34b0f086fbd77e6a610923d47363f8932b093c5d83d60da65f1b3f90 Jul 06 16:53:14.467338 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.467305 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-8mnwv"] Jul 06 16:53:14.470170 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.470152 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:14.473029 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.473007 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9djzt\"" Jul 06 16:53:14.473264 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.473249 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 16:53:14.489904 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.489868 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-8mnwv"] Jul 06 16:53:14.621155 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.621122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c04877-970a-411a-90de-5311266f5cc0-plugins-config-volume\") pod \"payload-pre-processing-7964457696-8mnwv\" (UID: \"71c04877-970a-411a-90de-5311266f5cc0\") " pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:14.621335 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.621193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcm4l\" (UniqueName: \"kubernetes.io/projected/71c04877-970a-411a-90de-5311266f5cc0-kube-api-access-mcm4l\") pod \"payload-pre-processing-7964457696-8mnwv\" (UID: \"71c04877-970a-411a-90de-5311266f5cc0\") " pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:14.722247 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.722154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c04877-970a-411a-90de-5311266f5cc0-plugins-config-volume\") pod \"payload-pre-processing-7964457696-8mnwv\" (UID: \"71c04877-970a-411a-90de-5311266f5cc0\") " pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:14.722247 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.722222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcm4l\" (UniqueName: \"kubernetes.io/projected/71c04877-970a-411a-90de-5311266f5cc0-kube-api-access-mcm4l\") pod \"payload-pre-processing-7964457696-8mnwv\" (UID: \"71c04877-970a-411a-90de-5311266f5cc0\") " pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:14.722999 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.722976 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c04877-970a-411a-90de-5311266f5cc0-plugins-config-volume\") pod \"payload-pre-processing-7964457696-8mnwv\" (UID: \"71c04877-970a-411a-90de-5311266f5cc0\") " pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:14.732179 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.732153 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcm4l\" (UniqueName: \"kubernetes.io/projected/71c04877-970a-411a-90de-5311266f5cc0-kube-api-access-mcm4l\") pod \"payload-pre-processing-7964457696-8mnwv\" (UID: \"71c04877-970a-411a-90de-5311266f5cc0\") " pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:14.780952 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.780915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:14.915914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:14.915888 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-8mnwv"] Jul 06 16:53:14.918533 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:53:14.918498 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c04877_970a_411a_90de_5311266f5cc0.slice/crio-b1df169732caee1394c6d6c699e8984e1c8f558b5f8a1ead3b046fbdbb5a427a WatchSource:0}: Error finding container b1df169732caee1394c6d6c699e8984e1c8f558b5f8a1ead3b046fbdbb5a427a: Status 404 returned error can't find the container with id b1df169732caee1394c6d6c699e8984e1c8f558b5f8a1ead3b046fbdbb5a427a Jul 06 16:53:15.314790 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:15.314752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" event={"ID":"71c04877-970a-411a-90de-5311266f5cc0","Type":"ContainerStarted","Data":"b1df169732caee1394c6d6c699e8984e1c8f558b5f8a1ead3b046fbdbb5a427a"} Jul 06 16:53:15.316019 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:15.315991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-78ddc7f974-4rtm2" event={"ID":"03c508e2-5ff7-46aa-9e63-c52efcb1128d","Type":"ContainerStarted","Data":"e2e08e3b34b0f086fbd77e6a610923d47363f8932b093c5d83d60da65f1b3f90"} Jul 06 16:53:16.322642 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:16.322593 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-78ddc7f974-4rtm2" event={"ID":"03c508e2-5ff7-46aa-9e63-c52efcb1128d","Type":"ContainerStarted","Data":"b51a63bdbc8ac688c479ec4e633059dac6e295acec9fdf7669cd8c2872f5a704"} Jul 06 16:53:16.323101 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:16.322731 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:16.342735 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:16.342663 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-78ddc7f974-4rtm2" podStartSLOduration=1.6067270420000002 podStartE2EDuration="3.342642258s" totalImagesPullingTime="1.735915216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:53:16.341998471 +0000 UTC m=+893.111890015" watchObservedRunningTime="2026-07-06 16:53:16.342642258 +0000 UTC m=+893.112533800" Jul 06 16:53:17.328960 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:17.328916 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" event={"ID":"71c04877-970a-411a-90de-5311266f5cc0","Type":"ContainerStarted","Data":"3dd0b20d3c03e77868ce53ff30f0477b080297b8a49d5d95598ed89c20e04bbb"} Jul 06 16:53:17.329501 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:17.329009 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:17.349088 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:17.349034 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" podStartSLOduration=1.747454456 podStartE2EDuration="3.349016061s" totalImagesPullingTime="1.601561605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:53:17.346486308 +0000 UTC m=+894.116377862" watchObservedRunningTime="2026-07-06 16:53:17.349016061 +0000 UTC m=+894.118907986" Jul 06 16:53:22.335002 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:22.334978 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-78ddc7f974-4rtm2" Jul 06 16:53:23.919434 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:23.919395 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:24.020591 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:24.020558 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:28.335651 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:28.335616 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-8mnwv" Jul 06 16:53:33.343368 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:33.343334 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:37.514947 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:37.514900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:53:59.125576 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:53:59.125534 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:54:08.807785 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.807750 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml"] Jul 06 16:54:08.810772 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.810746 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:08.813539 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.813518 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 16:54:08.814491 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.814471 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 16:54:08.814635 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.814607 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-s9bmb\"" Jul 06 16:54:08.814778 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.814528 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 16:54:08.824235 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.824205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml"] Jul 06 16:54:08.863188 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.863158 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:54:08.914977 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.914945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:08.914977 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.914978 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnz2\" (UniqueName: \"kubernetes.io/projected/13864aa8-03dc-4572-96a2-199bffc5a203-kube-api-access-rjnz2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:08.915189 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.915015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13864aa8-03dc-4572-96a2-199bffc5a203-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:08.915189 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.915088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:08.915189 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.915156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:08.915302 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:08.915224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016008 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.015964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016008 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.016007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnz2\" (UniqueName: \"kubernetes.io/projected/13864aa8-03dc-4572-96a2-199bffc5a203-kube-api-access-rjnz2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016237 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.016042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13864aa8-03dc-4572-96a2-199bffc5a203-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016237 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.016065 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016237 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.016110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016237 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.016144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016563 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.016542 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.016580 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.016638 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.016606 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.018450 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.018421 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13864aa8-03dc-4572-96a2-199bffc5a203-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.018633 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.018617 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13864aa8-03dc-4572-96a2-199bffc5a203-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.024090 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.024061 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnz2\" (UniqueName: \"kubernetes.io/projected/13864aa8-03dc-4572-96a2-199bffc5a203-kube-api-access-rjnz2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml\" (UID: \"13864aa8-03dc-4572-96a2-199bffc5a203\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.121700 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.121602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:09.259261 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.259235 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml"] Jul 06 16:54:09.262318 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:54:09.262284 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13864aa8_03dc_4572_96a2_199bffc5a203.slice/crio-901cb06ef0c88a690904317b8b98461498cd8ba2b9df4590c287d29703b876eb WatchSource:0}: Error finding container 901cb06ef0c88a690904317b8b98461498cd8ba2b9df4590c287d29703b876eb: Status 404 returned error can't find the container with id 901cb06ef0c88a690904317b8b98461498cd8ba2b9df4590c287d29703b876eb Jul 06 16:54:09.565635 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:09.565555 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" event={"ID":"13864aa8-03dc-4572-96a2-199bffc5a203","Type":"ContainerStarted","Data":"901cb06ef0c88a690904317b8b98461498cd8ba2b9df4590c287d29703b876eb"} Jul 06 16:54:10.571935 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:10.571838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" event={"ID":"13864aa8-03dc-4572-96a2-199bffc5a203","Type":"ContainerStarted","Data":"7b6f160906b95443119ad97493817b35daa31e76a7b2defc61d6296c063e9256"} Jul 06 16:54:10.598665 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:10.598604 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" podStartSLOduration=1.618686563 podStartE2EDuration="2.598585145s" totalImagesPullingTime="979.898582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:54:10.594107458 +0000 UTC m=+947.363998992" watchObservedRunningTime="2026-07-06 16:54:10.598585145 +0000 UTC m=+947.368476686" Jul 06 16:54:19.122481 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:19.122414 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:19.122909 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:19.122503 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:19.135242 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:19.135214 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:19.622585 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:19.622553 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml" Jul 06 16:54:20.973389 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:20.973354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:54:31.839662 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:54:31.839626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:55:31.591805 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:55:31.591758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:55:44.782797 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:55:44.782713 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:55:44.883762 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:55:44.883730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:56:05.075137 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:56:05.075079 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:56:05.174360 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:56:05.174331 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:56:06.218128 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:56:06.218094 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:56:06.353664 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:56:06.353629 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:56:58.567571 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:56:58.567516 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:57:16.651948 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:16.651918 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:57:16.795174 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:16.795143 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:57:16.897178 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:16.897135 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-589c49595b-82csr"] Jul 06 16:57:16.897425 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:16.897398 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-589c49595b-82csr" podUID="8f787297-0ba3-417a-90cc-d1866ebe0874" containerName="manager" containerID="cri-o://40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f" gracePeriod=30 Jul 06 16:57:17.151322 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.151293 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:57:17.224109 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.224032 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn979\" (UniqueName: \"kubernetes.io/projected/8f787297-0ba3-417a-90cc-d1866ebe0874-kube-api-access-tn979\") pod \"8f787297-0ba3-417a-90cc-d1866ebe0874\" (UID: \"8f787297-0ba3-417a-90cc-d1866ebe0874\") " Jul 06 16:57:17.224109 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.224074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f787297-0ba3-417a-90cc-d1866ebe0874-webhook-cert\") pod \"8f787297-0ba3-417a-90cc-d1866ebe0874\" (UID: \"8f787297-0ba3-417a-90cc-d1866ebe0874\") " Jul 06 16:57:17.226119 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.226080 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f787297-0ba3-417a-90cc-d1866ebe0874-kube-api-access-tn979" (OuterVolumeSpecName: "kube-api-access-tn979") pod "8f787297-0ba3-417a-90cc-d1866ebe0874" (UID: "8f787297-0ba3-417a-90cc-d1866ebe0874"). InnerVolumeSpecName "kube-api-access-tn979". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:57:17.226242 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.226131 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f787297-0ba3-417a-90cc-d1866ebe0874-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8f787297-0ba3-417a-90cc-d1866ebe0874" (UID: "8f787297-0ba3-417a-90cc-d1866ebe0874"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:57:17.325115 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.325077 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f787297-0ba3-417a-90cc-d1866ebe0874-webhook-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:57:17.325115 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.325108 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn979\" (UniqueName: \"kubernetes.io/projected/8f787297-0ba3-417a-90cc-d1866ebe0874-kube-api-access-tn979\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 16:57:17.403181 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.403141 2574 generic.go:356] "Generic (PLEG): container finished" podID="8f787297-0ba3-417a-90cc-d1866ebe0874" containerID="40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f" exitCode=0 Jul 06 16:57:17.403363 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.403196 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-589c49595b-82csr" Jul 06 16:57:17.403363 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.403211 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-589c49595b-82csr" event={"ID":"8f787297-0ba3-417a-90cc-d1866ebe0874","Type":"ContainerDied","Data":"40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f"} Jul 06 16:57:17.403363 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.403263 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-589c49595b-82csr" event={"ID":"8f787297-0ba3-417a-90cc-d1866ebe0874","Type":"ContainerDied","Data":"953583bfb2473b00b62a9bf44fdee16ed833833956782621c2ac76aaf1cc4ae1"} Jul 06 16:57:17.403363 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.403295 2574 scope.go:117] "RemoveContainer" containerID="40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f" Jul 06 16:57:17.412666 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.412641 2574 scope.go:117] "RemoveContainer" containerID="40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f" Jul 06 16:57:17.412990 ip-10-0-130-239 kubenswrapper[2574]: E0706 16:57:17.412971 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f\": container with ID starting with 40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f not found: ID does not exist" containerID="40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f" Jul 06 16:57:17.413049 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.413000 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f"} err="failed to get container status \"40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f\": rpc error: code = NotFound desc = could not find container \"40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f\": container with ID starting with 40417e4581dc2dc4f10d19637401fe5ee7055299c46e7ae1619f443dc2c6707f not found: ID does not exist" Jul 06 16:57:17.426359 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.426328 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-589c49595b-82csr"] Jul 06 16:57:17.429081 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.429050 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-589c49595b-82csr"] Jul 06 16:57:17.747423 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:17.747389 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f787297-0ba3-417a-90cc-d1866ebe0874" path="/var/lib/kubelet/pods/8f787297-0ba3-417a-90cc-d1866ebe0874/volumes" Jul 06 16:57:18.133281 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.133194 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:57:18.258369 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.258339 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-589c49595b-55qhp"] Jul 06 16:57:18.258737 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.258722 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f787297-0ba3-417a-90cc-d1866ebe0874" containerName="manager" Jul 06 16:57:18.258796 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.258739 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f787297-0ba3-417a-90cc-d1866ebe0874" containerName="manager" Jul 06 16:57:18.258830 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.258809 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f787297-0ba3-417a-90cc-d1866ebe0874" containerName="manager" Jul 06 16:57:18.262910 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.262892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:18.269491 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.269117 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 16:57:18.269491 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.269164 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b857d\"" Jul 06 16:57:18.269491 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.269285 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 16:57:18.278926 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.278890 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-589c49595b-55qhp"] Jul 06 16:57:18.333700 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.333664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61322c1f-7756-4293-a5c6-c2df5f79e4ed-webhook-cert\") pod \"maas-controller-589c49595b-55qhp\" (UID: \"61322c1f-7756-4293-a5c6-c2df5f79e4ed\") " pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:18.333700 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.333702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7thw\" (UniqueName: \"kubernetes.io/projected/61322c1f-7756-4293-a5c6-c2df5f79e4ed-kube-api-access-j7thw\") pod \"maas-controller-589c49595b-55qhp\" (UID: \"61322c1f-7756-4293-a5c6-c2df5f79e4ed\") " pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:18.435045 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.434948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61322c1f-7756-4293-a5c6-c2df5f79e4ed-webhook-cert\") pod \"maas-controller-589c49595b-55qhp\" (UID: \"61322c1f-7756-4293-a5c6-c2df5f79e4ed\") " pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:18.435045 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.434990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7thw\" (UniqueName: \"kubernetes.io/projected/61322c1f-7756-4293-a5c6-c2df5f79e4ed-kube-api-access-j7thw\") pod \"maas-controller-589c49595b-55qhp\" (UID: \"61322c1f-7756-4293-a5c6-c2df5f79e4ed\") " pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:18.437557 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.437534 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61322c1f-7756-4293-a5c6-c2df5f79e4ed-webhook-cert\") pod \"maas-controller-589c49595b-55qhp\" (UID: \"61322c1f-7756-4293-a5c6-c2df5f79e4ed\") " pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:18.443785 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.443758 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7thw\" (UniqueName: \"kubernetes.io/projected/61322c1f-7756-4293-a5c6-c2df5f79e4ed-kube-api-access-j7thw\") pod \"maas-controller-589c49595b-55qhp\" (UID: \"61322c1f-7756-4293-a5c6-c2df5f79e4ed\") " pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:18.572900 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.572862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:18.715914 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.715886 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-589c49595b-55qhp"] Jul 06 16:57:18.718182 ip-10-0-130-239 kubenswrapper[2574]: W0706 16:57:18.718154 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61322c1f_7756_4293_a5c6_c2df5f79e4ed.slice/crio-edb39c38acf56ec5f4d99ce8cd4d15faac5f868651167d5793d4716cf0ef3bba WatchSource:0}: Error finding container edb39c38acf56ec5f4d99ce8cd4d15faac5f868651167d5793d4716cf0ef3bba: Status 404 returned error can't find the container with id edb39c38acf56ec5f4d99ce8cd4d15faac5f868651167d5793d4716cf0ef3bba Jul 06 16:57:18.719573 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:18.719555 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:57:19.414672 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:19.414633 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-589c49595b-55qhp" event={"ID":"61322c1f-7756-4293-a5c6-c2df5f79e4ed","Type":"ContainerStarted","Data":"edb39c38acf56ec5f4d99ce8cd4d15faac5f868651167d5793d4716cf0ef3bba"} Jul 06 16:57:20.420560 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:20.420524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-589c49595b-55qhp" event={"ID":"61322c1f-7756-4293-a5c6-c2df5f79e4ed","Type":"ContainerStarted","Data":"1417f994f62b542f72e91df2c7f93caa3fa8fbcdb5a2dbd062f213358223c996"} Jul 06 16:57:20.420961 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:20.420655 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:20.455075 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:20.455026 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-589c49595b-55qhp" podStartSLOduration=1.3928097400000001 podStartE2EDuration="2.455006977s" totalImagesPullingTime="1.062197237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:57:20.454833829 +0000 UTC m=+1137.224725372" watchObservedRunningTime="2026-07-06 16:57:20.455006977 +0000 UTC m=+1137.224898518" Jul 06 16:57:31.431560 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:31.431527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-589c49595b-55qhp" Jul 06 16:57:35.459381 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:35.459342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:57:35.523105 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:35.523068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:57:58.113083 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:58.113048 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:57:58.210109 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:58.210063 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:57:58.334050 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:57:58.334002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:04.685930 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:04.685893 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:04.750781 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:04.750741 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:04.828071 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:04.828036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:04.925408 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:04.925375 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:05.053641 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:05.053612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:11.294907 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:11.294861 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:11.388178 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:11.388142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:45.494207 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:45.494173 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:58:45.564047 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:58:45.564009 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 16:59:57.568435 ip-10-0-130-239 kubenswrapper[2574]: I0706 16:59:57.568394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:00:00.157630 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:00.157587 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722620-fqnlp"] Jul 06 17:00:00.161164 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:00.161140 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" Jul 06 17:00:00.187400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:00.187352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722620-fqnlp"] Jul 06 17:00:00.215074 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:00.215036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zczqg\" (UniqueName: \"kubernetes.io/projected/539fd698-c3a4-4a12-80bd-87e52e2c7246-kube-api-access-zczqg\") pod \"maas-api-key-cleanup-29722620-fqnlp\" (UID: \"539fd698-c3a4-4a12-80bd-87e52e2c7246\") " pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" Jul 06 17:00:00.315734 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:00.315689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zczqg\" (UniqueName: \"kubernetes.io/projected/539fd698-c3a4-4a12-80bd-87e52e2c7246-kube-api-access-zczqg\") pod \"maas-api-key-cleanup-29722620-fqnlp\" (UID: \"539fd698-c3a4-4a12-80bd-87e52e2c7246\") " pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" Jul 06 17:00:00.327638 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:00.327599 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zczqg\" (UniqueName: \"kubernetes.io/projected/539fd698-c3a4-4a12-80bd-87e52e2c7246-kube-api-access-zczqg\") pod \"maas-api-key-cleanup-29722620-fqnlp\" (UID: \"539fd698-c3a4-4a12-80bd-87e52e2c7246\") " pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" Jul 06 17:00:00.472092 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:00.471985 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" Jul 06 17:00:00.613524 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:00.613439 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722620-fqnlp"] Jul 06 17:00:00.617511 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:00:00.617448 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539fd698_c3a4_4a12_80bd_87e52e2c7246.slice/crio-b0615ab6bd4234d417720c2085f91d5dc1ad53d9ffa1bb828da928c5582dc4b1 WatchSource:0}: Error finding container b0615ab6bd4234d417720c2085f91d5dc1ad53d9ffa1bb828da928c5582dc4b1: Status 404 returned error can't find the container with id b0615ab6bd4234d417720c2085f91d5dc1ad53d9ffa1bb828da928c5582dc4b1 Jul 06 17:00:01.156410 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:01.156373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" event={"ID":"539fd698-c3a4-4a12-80bd-87e52e2c7246","Type":"ContainerStarted","Data":"b0615ab6bd4234d417720c2085f91d5dc1ad53d9ffa1bb828da928c5582dc4b1"} Jul 06 17:00:12.208752 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:12.208714 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" event={"ID":"539fd698-c3a4-4a12-80bd-87e52e2c7246","Type":"ContainerStarted","Data":"4d6f54edc18a56c615d16597a3a46b4ca468938ed7f633014ab283636063b8a5"} Jul 06 17:00:12.238758 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:12.238706 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" podStartSLOduration=1.109476197 podStartE2EDuration="12.238687823s" totalImagesPullingTime="11.129211626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:00:12.238508778 +0000 UTC m=+1309.008400319" watchObservedRunningTime="2026-07-06 17:00:12.238687823 +0000 UTC m=+1309.008579364" Jul 06 17:00:13.214595 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:13.214556 2574 generic.go:356] "Generic (PLEG): container finished" podID="539fd698-c3a4-4a12-80bd-87e52e2c7246" containerID="4d6f54edc18a56c615d16597a3a46b4ca468938ed7f633014ab283636063b8a5" exitCode=0 Jul 06 17:00:13.214595 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:13.214597 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" event={"ID":"539fd698-c3a4-4a12-80bd-87e52e2c7246","Type":"ContainerDied","Data":"4d6f54edc18a56c615d16597a3a46b4ca468938ed7f633014ab283636063b8a5"} Jul 06 17:00:14.356764 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:14.356738 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" Jul 06 17:00:14.448342 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:14.448308 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zczqg\" (UniqueName: \"kubernetes.io/projected/539fd698-c3a4-4a12-80bd-87e52e2c7246-kube-api-access-zczqg\") pod \"539fd698-c3a4-4a12-80bd-87e52e2c7246\" (UID: \"539fd698-c3a4-4a12-80bd-87e52e2c7246\") " Jul 06 17:00:14.450391 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:14.450360 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539fd698-c3a4-4a12-80bd-87e52e2c7246-kube-api-access-zczqg" (OuterVolumeSpecName: "kube-api-access-zczqg") pod "539fd698-c3a4-4a12-80bd-87e52e2c7246" (UID: "539fd698-c3a4-4a12-80bd-87e52e2c7246"). InnerVolumeSpecName "kube-api-access-zczqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:00:14.549650 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:14.549609 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zczqg\" (UniqueName: \"kubernetes.io/projected/539fd698-c3a4-4a12-80bd-87e52e2c7246-kube-api-access-zczqg\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:00:15.225803 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:15.225776 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" Jul 06 17:00:15.225984 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:15.225764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722620-fqnlp" event={"ID":"539fd698-c3a4-4a12-80bd-87e52e2c7246","Type":"ContainerDied","Data":"b0615ab6bd4234d417720c2085f91d5dc1ad53d9ffa1bb828da928c5582dc4b1"} Jul 06 17:00:15.225984 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:00:15.225897 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0615ab6bd4234d417720c2085f91d5dc1ad53d9ffa1bb828da928c5582dc4b1" Jul 06 17:01:53.025181 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:01:53.025139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:01:54.557033 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:01:54.556974 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:05:18.703981 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:05:18.703899 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:07:42.041963 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:07:42.041927 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:08:47.518635 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:08:47.518598 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:08:52.846973 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:08:52.846938 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:01.100724 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:01.100678 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:06.201351 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:06.201314 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:09.010394 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:09.010355 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:09.108905 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:09.108872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:09.477502 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:09.477389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:15.884369 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:15.884334 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:15.959760 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:15.959730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:16.106789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:16.106752 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:16.203709 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:16.203622 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:17.294697 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.294663 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:17.409488 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.409411 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:17.412323 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.412296 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx"] Jul 06 17:09:17.412689 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.412676 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539fd698-c3a4-4a12-80bd-87e52e2c7246" containerName="cleanup" Jul 06 17:09:17.412737 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.412692 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="539fd698-c3a4-4a12-80bd-87e52e2c7246" containerName="cleanup" Jul 06 17:09:17.412783 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.412772 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="539fd698-c3a4-4a12-80bd-87e52e2c7246" containerName="cleanup" Jul 06 17:09:17.415630 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.415613 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.418078 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.418057 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-38d19e47-serving-cert\"" Jul 06 17:09:17.448383 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.448352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx"] Jul 06 17:09:17.496561 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.496523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx7zz\" (UniqueName: \"kubernetes.io/projected/fd8754b2-1cf5-4571-b9ab-76b81e099f58-kube-api-access-nx7zz\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.496732 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.496679 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8754b2-1cf5-4571-b9ab-76b81e099f58-maas-api-tls\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.496784 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.496733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8754b2-1cf5-4571-b9ab-76b81e099f58-openshift-service-ca\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.512247 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.512216 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:17.597825 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.597734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8754b2-1cf5-4571-b9ab-76b81e099f58-maas-api-tls\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.597825 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.597812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8754b2-1cf5-4571-b9ab-76b81e099f58-openshift-service-ca\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.598008 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.597851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx7zz\" (UniqueName: \"kubernetes.io/projected/fd8754b2-1cf5-4571-b9ab-76b81e099f58-kube-api-access-nx7zz\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.598543 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.598521 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8754b2-1cf5-4571-b9ab-76b81e099f58-openshift-service-ca\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.600164 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.600142 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8754b2-1cf5-4571-b9ab-76b81e099f58-maas-api-tls\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.615809 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.615783 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx7zz\" (UniqueName: \"kubernetes.io/projected/fd8754b2-1cf5-4571-b9ab-76b81e099f58-kube-api-access-nx7zz\") pod \"maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.657246 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.657209 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:17.726597 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.726556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:17.860650 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.860625 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx"] Jul 06 17:09:17.863117 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:09:17.863085 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8754b2_1cf5_4571_b9ab_76b81e099f58.slice/crio-31a248e62ef52b907c4e8a30c6c7bc95c81c19e2b32d3f859165acefda4f9262 WatchSource:0}: Error finding container 31a248e62ef52b907c4e8a30c6c7bc95c81c19e2b32d3f859165acefda4f9262: Status 404 returned error can't find the container with id 31a248e62ef52b907c4e8a30c6c7bc95c81c19e2b32d3f859165acefda4f9262 Jul 06 17:09:17.864503 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:17.864481 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 17:09:18.082084 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:18.082055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:18.178555 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:18.178475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:18.708552 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:18.708511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" event={"ID":"fd8754b2-1cf5-4571-b9ab-76b81e099f58","Type":"ContainerStarted","Data":"3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe"} Jul 06 17:09:18.708957 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:18.708563 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" event={"ID":"fd8754b2-1cf5-4571-b9ab-76b81e099f58","Type":"ContainerStarted","Data":"31a248e62ef52b907c4e8a30c6c7bc95c81c19e2b32d3f859165acefda4f9262"} Jul 06 17:09:18.708957 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:18.708631 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:18.726067 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:18.726024 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" podStartSLOduration=1.051756987 podStartE2EDuration="1.726007027s" totalImagesPullingTime="674.25004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:18.725132354 +0000 UTC m=+1855.495023895" watchObservedRunningTime="2026-07-06 17:09:18.726007027 +0000 UTC m=+1855.495898618" Jul 06 17:09:19.833690 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:19.833661 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:22.402409 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:22.402374 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:22.652986 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:22.652901 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:22.753321 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:22.753287 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:22.771184 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:22.771150 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx"] Jul 06 17:09:22.771419 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:22.771391 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" podUID="fd8754b2-1cf5-4571-b9ab-76b81e099f58" containerName="maas-api" containerID="cri-o://3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe" gracePeriod=30 Jul 06 17:09:22.816292 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:22.816260 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:23.021995 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.021962 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:23.149813 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.149782 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8754b2-1cf5-4571-b9ab-76b81e099f58-maas-api-tls\") pod \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " Jul 06 17:09:23.149981 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.149827 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx7zz\" (UniqueName: \"kubernetes.io/projected/fd8754b2-1cf5-4571-b9ab-76b81e099f58-kube-api-access-nx7zz\") pod \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " Jul 06 17:09:23.149981 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.149888 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8754b2-1cf5-4571-b9ab-76b81e099f58-openshift-service-ca\") pod \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\" (UID: \"fd8754b2-1cf5-4571-b9ab-76b81e099f58\") " Jul 06 17:09:23.150285 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.150257 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8754b2-1cf5-4571-b9ab-76b81e099f58-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd8754b2-1cf5-4571-b9ab-76b81e099f58" (UID: "fd8754b2-1cf5-4571-b9ab-76b81e099f58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:09:23.151839 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.151815 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8754b2-1cf5-4571-b9ab-76b81e099f58-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd8754b2-1cf5-4571-b9ab-76b81e099f58" (UID: "fd8754b2-1cf5-4571-b9ab-76b81e099f58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:09:23.151922 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.151875 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8754b2-1cf5-4571-b9ab-76b81e099f58-kube-api-access-nx7zz" (OuterVolumeSpecName: "kube-api-access-nx7zz") pod "fd8754b2-1cf5-4571-b9ab-76b81e099f58" (UID: "fd8754b2-1cf5-4571-b9ab-76b81e099f58"). InnerVolumeSpecName "kube-api-access-nx7zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:09:23.251396 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.251360 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd8754b2-1cf5-4571-b9ab-76b81e099f58-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:23.251396 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.251390 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx7zz\" (UniqueName: \"kubernetes.io/projected/fd8754b2-1cf5-4571-b9ab-76b81e099f58-kube-api-access-nx7zz\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:23.251396 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.251399 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8754b2-1cf5-4571-b9ab-76b81e099f58-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:23.734579 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.734549 2574 generic.go:356] "Generic (PLEG): container finished" podID="fd8754b2-1cf5-4571-b9ab-76b81e099f58" containerID="3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe" exitCode=0 Jul 06 17:09:23.734955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.734597 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" Jul 06 17:09:23.734955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.734641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" event={"ID":"fd8754b2-1cf5-4571-b9ab-76b81e099f58","Type":"ContainerDied","Data":"3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe"} Jul 06 17:09:23.734955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.734695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx" event={"ID":"fd8754b2-1cf5-4571-b9ab-76b81e099f58","Type":"ContainerDied","Data":"31a248e62ef52b907c4e8a30c6c7bc95c81c19e2b32d3f859165acefda4f9262"} Jul 06 17:09:23.734955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.734719 2574 scope.go:117] "RemoveContainer" containerID="3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe" Jul 06 17:09:23.744079 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.744055 2574 scope.go:117] "RemoveContainer" containerID="3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe" Jul 06 17:09:23.744344 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:09:23.744310 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe\": container with ID starting with 3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe not found: ID does not exist" containerID="3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe" Jul 06 17:09:23.744394 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.744366 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe"} err="failed to get container status \"3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe\": rpc error: code = NotFound desc = could not find container \"3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe\": container with ID starting with 3a36fa1912032beedd922db95d3a5cf1a59735491a520d5faac312dd11611cfe not found: ID does not exist" Jul 06 17:09:23.758913 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.758888 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx"] Jul 06 17:09:23.762700 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:23.762680 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-38d19e47-96f49d5bb-hfzmx"] Jul 06 17:09:24.179111 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:24.179082 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:25.747932 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:25.747889 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd8754b2-1cf5-4571-b9ab-76b81e099f58" path="/var/lib/kubelet/pods/fd8754b2-1cf5-4571-b9ab-76b81e099f58/volumes" Jul 06 17:09:29.038055 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.038025 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:29.139981 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.139952 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:29.230253 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.230217 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48"] Jul 06 17:09:29.230609 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.230596 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8754b2-1cf5-4571-b9ab-76b81e099f58" containerName="maas-api" Jul 06 17:09:29.230609 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.230611 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8754b2-1cf5-4571-b9ab-76b81e099f58" containerName="maas-api" Jul 06 17:09:29.230708 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.230685 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8754b2-1cf5-4571-b9ab-76b81e099f58" containerName="maas-api" Jul 06 17:09:29.233696 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.233676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.236239 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.236218 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5038f666-openshift-default-dockercfg-btw9b\"" Jul 06 17:09:29.246941 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.246914 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48"] Jul 06 17:09:29.270241 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.270215 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:29.375654 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.375583 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:29.411693 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a419a-7253-4198-b7dc-dd23bc58920b-istio-podinfo\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.411847 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-socket\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.411847 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-data\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.411847 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-istio-token\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.411948 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-credential-socket\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.411948 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a419a-7253-4198-b7dc-dd23bc58920b-istiod-ca-cert\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.411948 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh7f4\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-kube-api-access-fh7f4\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.411948 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-envoy\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.412103 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.411950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-certs\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.512616 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a419a-7253-4198-b7dc-dd23bc58920b-istiod-ca-cert\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.512616 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh7f4\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-kube-api-access-fh7f4\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.512823 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-envoy\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.512823 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-certs\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.512823 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a419a-7253-4198-b7dc-dd23bc58920b-istio-podinfo\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.512823 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-socket\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.512823 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-data\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.513064 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-istio-token\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.513064 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.512886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-credential-socket\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.513170 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.513121 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-certs\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.513248 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.513218 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-socket\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.513409 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.513379 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-data\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.513514 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.513415 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a419a-7253-4198-b7dc-dd23bc58920b-istiod-ca-cert\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.513563 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.513516 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-credential-socket\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.515171 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.515151 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-envoy\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.515275 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.515257 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a419a-7253-4198-b7dc-dd23bc58920b-istio-podinfo\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.522442 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.522416 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-istio-token\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.522943 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.522917 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh7f4\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-kube-api-access-fh7f4\") pod \"e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.544961 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.544923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:29.687525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.687501 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48"] Jul 06 17:09:29.689764 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:09:29.689735 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079a419a_7253_4198_b7dc_dd23bc58920b.slice/crio-8c2f045e049f09e730967086cac5a3a0efa902106ca71cb7f3f1f7ad1776acb8 WatchSource:0}: Error finding container 8c2f045e049f09e730967086cac5a3a0efa902106ca71cb7f3f1f7ad1776acb8: Status 404 returned error can't find the container with id 8c2f045e049f09e730967086cac5a3a0efa902106ca71cb7f3f1f7ad1776acb8 Jul 06 17:09:29.760907 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:29.760869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" event={"ID":"079a419a-7253-4198-b7dc-dd23bc58920b","Type":"ContainerStarted","Data":"8c2f045e049f09e730967086cac5a3a0efa902106ca71cb7f3f1f7ad1776acb8"} Jul 06 17:09:30.498469 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.498422 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:30.585477 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.585415 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd"] Jul 06 17:09:30.589007 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.588977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.592027 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.592001 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-5038f666-serving-cert\"" Jul 06 17:09:30.605336 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.605304 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd"] Jul 06 17:09:30.644977 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.644945 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:30.725013 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.724978 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtt4\" (UniqueName: \"kubernetes.io/projected/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-kube-api-access-jhtt4\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.725204 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.725049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-maas-api-tls\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.725204 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.725092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-openshift-service-ca\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.826325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.826245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtt4\" (UniqueName: \"kubernetes.io/projected/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-kube-api-access-jhtt4\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.826325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.826316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-maas-api-tls\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.826541 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.826354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-openshift-service-ca\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.827109 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.827084 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-openshift-service-ca\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.829328 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.829299 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-maas-api-tls\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.835482 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.835442 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtt4\" (UniqueName: \"kubernetes.io/projected/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-kube-api-access-jhtt4\") pod \"maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.903176 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.903140 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:30.906729 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:30.906704 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:31.073092 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:31.073068 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd"] Jul 06 17:09:31.076559 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:09:31.076488 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5528f1a_400b_4efb_896e_5ca5b61ee9e9.slice/crio-6ef0465064a0cb1faed8b8b7917175b4f545eb3b5a538e8596dfd1c3d3778907 WatchSource:0}: Error finding container 6ef0465064a0cb1faed8b8b7917175b4f545eb3b5a538e8596dfd1c3d3778907: Status 404 returned error can't find the container with id 6ef0465064a0cb1faed8b8b7917175b4f545eb3b5a538e8596dfd1c3d3778907 Jul 06 17:09:31.120481 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:31.120428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:31.227035 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:31.226970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:31.771752 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:31.771705 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" event={"ID":"c5528f1a-400b-4efb-896e-5ca5b61ee9e9","Type":"ContainerStarted","Data":"6ef0465064a0cb1faed8b8b7917175b4f545eb3b5a538e8596dfd1c3d3778907"} Jul 06 17:09:32.440378 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.440344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:32.487074 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.486961 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:09:32.487074 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.487042 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:09:32.487169 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.487073 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:09:32.778734 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.778693 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" event={"ID":"079a419a-7253-4198-b7dc-dd23bc58920b","Type":"ContainerStarted","Data":"6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f"} Jul 06 17:09:32.780284 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.780250 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" event={"ID":"c5528f1a-400b-4efb-896e-5ca5b61ee9e9","Type":"ContainerStarted","Data":"94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3"} Jul 06 17:09:32.780388 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.780312 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:32.803997 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.803947 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" podStartSLOduration=1.009409921 podStartE2EDuration="3.803929669s" totalImagesPullingTime="2.794519748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:32.80308595 +0000 UTC m=+1869.572977515" watchObservedRunningTime="2026-07-06 17:09:32.803929669 +0000 UTC m=+1869.573821210" Jul 06 17:09:32.821679 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:32.821629 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" podStartSLOduration=1.413540493 podStartE2EDuration="2.821613056s" totalImagesPullingTime="1.408072563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:32.820123963 +0000 UTC m=+1869.590015505" watchObservedRunningTime="2026-07-06 17:09:32.821613056 +0000 UTC m=+1869.591504604" Jul 06 17:09:33.546061 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:33.546016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:33.551040 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:33.551009 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:33.785186 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:33.785146 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:33.827592 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:33.827494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:35.689857 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:35.689824 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:36.021445 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.021406 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd"] Jul 06 17:09:36.021719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.021687 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" podUID="c5528f1a-400b-4efb-896e-5ca5b61ee9e9" containerName="maas-api" containerID="cri-o://94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3" gracePeriod=30 Jul 06 17:09:36.057716 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.057680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:36.157126 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.157066 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:36.307950 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.307925 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:36.482011 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.481973 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-openshift-service-ca\") pod \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " Jul 06 17:09:36.482186 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.482060 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-maas-api-tls\") pod \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " Jul 06 17:09:36.482186 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.482141 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhtt4\" (UniqueName: \"kubernetes.io/projected/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-kube-api-access-jhtt4\") pod \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\" (UID: \"c5528f1a-400b-4efb-896e-5ca5b61ee9e9\") " Jul 06 17:09:36.482341 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.482311 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5528f1a-400b-4efb-896e-5ca5b61ee9e9" (UID: "c5528f1a-400b-4efb-896e-5ca5b61ee9e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:09:36.482480 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.482437 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:36.484150 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.484125 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5528f1a-400b-4efb-896e-5ca5b61ee9e9" (UID: "c5528f1a-400b-4efb-896e-5ca5b61ee9e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:09:36.484256 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.484195 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-kube-api-access-jhtt4" (OuterVolumeSpecName: "kube-api-access-jhtt4") pod "c5528f1a-400b-4efb-896e-5ca5b61ee9e9" (UID: "c5528f1a-400b-4efb-896e-5ca5b61ee9e9"). InnerVolumeSpecName "kube-api-access-jhtt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:09:36.583330 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.583254 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:36.583330 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.583281 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtt4\" (UniqueName: \"kubernetes.io/projected/c5528f1a-400b-4efb-896e-5ca5b61ee9e9-kube-api-access-jhtt4\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:36.806235 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.806192 2574 generic.go:356] "Generic (PLEG): container finished" podID="c5528f1a-400b-4efb-896e-5ca5b61ee9e9" containerID="94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3" exitCode=0 Jul 06 17:09:36.806674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.806260 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" Jul 06 17:09:36.806674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.806261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" event={"ID":"c5528f1a-400b-4efb-896e-5ca5b61ee9e9","Type":"ContainerDied","Data":"94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3"} Jul 06 17:09:36.806674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.806343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd" event={"ID":"c5528f1a-400b-4efb-896e-5ca5b61ee9e9","Type":"ContainerDied","Data":"6ef0465064a0cb1faed8b8b7917175b4f545eb3b5a538e8596dfd1c3d3778907"} Jul 06 17:09:36.806674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.806373 2574 scope.go:117] "RemoveContainer" containerID="94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3" Jul 06 17:09:36.815738 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.815717 2574 scope.go:117] "RemoveContainer" containerID="94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3" Jul 06 17:09:36.816009 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:09:36.815990 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3\": container with ID starting with 94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3 not found: ID does not exist" containerID="94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3" Jul 06 17:09:36.816089 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.816017 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3"} err="failed to get container status \"94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3\": rpc error: code = NotFound desc = could not find container \"94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3\": container with ID starting with 94c25eef964c6dbd4cd0078ce1ad62bc035071f40771481341a97c1262c2ddb3 not found: ID does not exist" Jul 06 17:09:36.839744 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.839631 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd"] Jul 06 17:09:36.845842 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:36.845810 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-5038f666-667df4f5c7-kkqmd"] Jul 06 17:09:37.752527 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:37.752493 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5528f1a-400b-4efb-896e-5ca5b61ee9e9" path="/var/lib/kubelet/pods/c5528f1a-400b-4efb-896e-5ca5b61ee9e9/volumes" Jul 06 17:09:40.825609 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:40.825580 2574 generic.go:356] "Generic (PLEG): container finished" podID="079a419a-7253-4198-b7dc-dd23bc58920b" containerID="6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f" exitCode=0 Jul 06 17:09:40.826055 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:40.825666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" event={"ID":"079a419a-7253-4198-b7dc-dd23bc58920b","Type":"ContainerDied","Data":"6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f"} Jul 06 17:09:40.826181 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:40.826161 2574 scope.go:117] "RemoveContainer" containerID="6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f" Jul 06 17:09:40.828078 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:40.828047 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:09:40.828173 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:40.828118 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:09:40.828173 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:40.828152 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:09:41.834698 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:41.834653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" event={"ID":"079a419a-7253-4198-b7dc-dd23bc58920b","Type":"ContainerStarted","Data":"aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba"} Jul 06 17:09:42.000479 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:42.000436 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48"] Jul 06 17:09:42.022379 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:42.022347 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:42.130914 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:42.130818 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:42.546082 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:42.546044 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:42.546493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:42.546390 2574 patch_prober.go:28] interesting pod/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 06 17:09:42.546712 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:42.546686 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 06 17:09:42.841579 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:42.841404 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" containerName="istio-proxy" containerID="cri-o://aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba" gracePeriod=30 Jul 06 17:09:44.364543 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:44.364512 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:48.613039 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:48.612958 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:48.701717 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:48.701682 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:48.785477 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:48.785405 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:48.880372 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:48.880301 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:48.976023 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:48.975980 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:50.192519 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.192478 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:50.270384 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.270350 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65"] Jul 06 17:09:50.270763 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.270750 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5528f1a-400b-4efb-896e-5ca5b61ee9e9" containerName="maas-api" Jul 06 17:09:50.270810 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.270765 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5528f1a-400b-4efb-896e-5ca5b61ee9e9" containerName="maas-api" Jul 06 17:09:50.270851 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.270820 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5528f1a-400b-4efb-896e-5ca5b61ee9e9" containerName="maas-api" Jul 06 17:09:50.275375 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.275359 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.278400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.278372 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-8b33da08-serving-cert\"" Jul 06 17:09:50.279156 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.279132 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:50.292095 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.292071 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65"] Jul 06 17:09:50.326882 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.326844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.327035 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.326892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shhg\" (UniqueName: \"kubernetes.io/projected/064812b2-c0e2-4071-8a08-ab1bb95c77bb-kube-api-access-7shhg\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.327075 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.327039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064812b2-c0e2-4071-8a08-ab1bb95c77bb-openshift-service-ca\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.347449 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.347419 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:50.428148 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.428115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064812b2-c0e2-4071-8a08-ab1bb95c77bb-openshift-service-ca\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.428405 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.428166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.428405 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.428189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shhg\" (UniqueName: \"kubernetes.io/projected/064812b2-c0e2-4071-8a08-ab1bb95c77bb-kube-api-access-7shhg\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.428405 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:09:50.428339 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-8b33da08-serving-cert: secret "maas-api-e2e-derive-8b33da08-serving-cert" not found Jul 06 17:09:50.428614 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:09:50.428410 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls podName:064812b2-c0e2-4071-8a08-ab1bb95c77bb nodeName:}" failed. No retries permitted until 2026-07-06 17:09:50.928390769 +0000 UTC m=+1887.698282327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls") pod "maas-api-e2e-derive-8b33da08-59565bc79-wft65" (UID: "064812b2-c0e2-4071-8a08-ab1bb95c77bb") : secret "maas-api-e2e-derive-8b33da08-serving-cert" not found Jul 06 17:09:50.428798 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.428780 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064812b2-c0e2-4071-8a08-ab1bb95c77bb-openshift-service-ca\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.437503 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.437482 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shhg\" (UniqueName: \"kubernetes.io/projected/064812b2-c0e2-4071-8a08-ab1bb95c77bb-kube-api-access-7shhg\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.698105 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.698075 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:50.801133 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.801107 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:50.826243 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.826205 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:50.883918 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.883830 2574 generic.go:356] "Generic (PLEG): container finished" podID="079a419a-7253-4198-b7dc-dd23bc58920b" containerID="aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba" exitCode=0 Jul 06 17:09:50.883918 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.883869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" event={"ID":"079a419a-7253-4198-b7dc-dd23bc58920b","Type":"ContainerDied","Data":"aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba"} Jul 06 17:09:50.883918 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.883910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" event={"ID":"079a419a-7253-4198-b7dc-dd23bc58920b","Type":"ContainerDied","Data":"8c2f045e049f09e730967086cac5a3a0efa902106ca71cb7f3f1f7ad1776acb8"} Jul 06 17:09:50.883918 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.883911 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48" Jul 06 17:09:50.884230 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.883932 2574 scope.go:117] "RemoveContainer" containerID="aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba" Jul 06 17:09:50.892546 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.892525 2574 scope.go:117] "RemoveContainer" containerID="6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f" Jul 06 17:09:50.900176 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.900159 2574 scope.go:117] "RemoveContainer" containerID="aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba" Jul 06 17:09:50.900476 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:09:50.900446 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba\": container with ID starting with aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba not found: ID does not exist" containerID="aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba" Jul 06 17:09:50.900535 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.900487 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba"} err="failed to get container status \"aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba\": rpc error: code = NotFound desc = could not find container \"aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba\": container with ID starting with aa6369a1701e48df0df61d2330f26fea36666e3e8c0e5cbd0d83f0a7382edfba not found: ID does not exist" Jul 06 17:09:50.900535 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.900505 2574 scope.go:117] "RemoveContainer" containerID="6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f" Jul 06 17:09:50.900759 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:09:50.900742 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f\": container with ID starting with 6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f not found: ID does not exist" containerID="6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f" Jul 06 17:09:50.900801 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.900765 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f"} err="failed to get container status \"6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f\": rpc error: code = NotFound desc = could not find container \"6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f\": container with ID starting with 6b142a737cd6aab0952d669a21a441c2666c970a27ca49a99287e60e41c8e92f not found: ID does not exist" Jul 06 17:09:50.927280 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.927248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:50.932304 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932285 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a419a-7253-4198-b7dc-dd23bc58920b-istio-podinfo\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932367 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932333 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-socket\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932367 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932360 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-istio-token\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932442 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932377 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a419a-7253-4198-b7dc-dd23bc58920b-istiod-ca-cert\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932442 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932408 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-envoy\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932442 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932428 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-data\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932992 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932494 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh7f4\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-kube-api-access-fh7f4\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932992 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932512 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-certs\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932992 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932549 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-credential-socket\") pod \"079a419a-7253-4198-b7dc-dd23bc58920b\" (UID: \"079a419a-7253-4198-b7dc-dd23bc58920b\") " Jul 06 17:09:50.932992 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.932992 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932722 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:50.932992 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932816 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079a419a-7253-4198-b7dc-dd23bc58920b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:09:50.933381 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.933003 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:50.933381 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.932942 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:50.933381 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.933041 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a419a-7253-4198-b7dc-dd23bc58920b-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:50.933381 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.933179 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:50.933381 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.933289 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-data" (OuterVolumeSpecName: "istio-data") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:50.935133 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.935090 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-kube-api-access-fh7f4" (OuterVolumeSpecName: "kube-api-access-fh7f4") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "kube-api-access-fh7f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:09:50.935133 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.935098 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-istio-token" (OuterVolumeSpecName: "istio-token") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:09:50.935294 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.935157 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls\") pod \"maas-api-e2e-derive-8b33da08-59565bc79-wft65\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:50.935334 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.935320 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/079a419a-7253-4198-b7dc-dd23bc58920b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:09:50.935399 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:50.935380 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "079a419a-7253-4198-b7dc-dd23bc58920b" (UID: "079a419a-7253-4198-b7dc-dd23bc58920b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:51.034443 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.034404 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh7f4\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-kube-api-access-fh7f4\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:51.034443 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.034437 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:51.034674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.034450 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:51.034674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.034488 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a419a-7253-4198-b7dc-dd23bc58920b-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:51.034674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.034499 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a419a-7253-4198-b7dc-dd23bc58920b-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:51.034674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.034509 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:51.034674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.034519 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a419a-7253-4198-b7dc-dd23bc58920b-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:51.185143 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.185053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:51.208510 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.208484 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48"] Jul 06 17:09:51.215314 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.215287 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5038f666-openshift-default-6599cb77dc-2hq48"] Jul 06 17:09:51.323292 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.323229 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65"] Jul 06 17:09:51.325739 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:09:51.325711 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064812b2_c0e2_4071_8a08_ab1bb95c77bb.slice/crio-1fea554ec9997d715b115d9de787710e4793e3ffa29c7fe78c1a61508e437c2e WatchSource:0}: Error finding container 1fea554ec9997d715b115d9de787710e4793e3ffa29c7fe78c1a61508e437c2e: Status 404 returned error can't find the container with id 1fea554ec9997d715b115d9de787710e4793e3ffa29c7fe78c1a61508e437c2e Jul 06 17:09:51.751009 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.750981 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" path="/var/lib/kubelet/pods/079a419a-7253-4198-b7dc-dd23bc58920b/volumes" Jul 06 17:09:51.890017 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.889981 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" event={"ID":"064812b2-c0e2-4071-8a08-ab1bb95c77bb","Type":"ContainerStarted","Data":"98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c"} Jul 06 17:09:51.890240 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.890061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" event={"ID":"064812b2-c0e2-4071-8a08-ab1bb95c77bb","Type":"ContainerStarted","Data":"1fea554ec9997d715b115d9de787710e4793e3ffa29c7fe78c1a61508e437c2e"} Jul 06 17:09:51.890240 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.890092 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:51.909102 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:51.909052 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" podStartSLOduration=1.5424221390000001 podStartE2EDuration="1.909037214s" totalImagesPullingTime="366.615075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:51.907333803 +0000 UTC m=+1888.677225353" watchObservedRunningTime="2026-07-06 17:09:51.909037214 +0000 UTC m=+1888.678928795" Jul 06 17:09:52.205136 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:52.205113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:54.605419 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:54.605370 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:54.842963 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:54.842927 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:54.954150 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:54.954055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:55.053377 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.053347 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65"] Jul 06 17:09:55.053592 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.053569 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" podUID="064812b2-c0e2-4071-8a08-ab1bb95c77bb" containerName="maas-api" containerID="cri-o://98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c" gracePeriod=30 Jul 06 17:09:55.077175 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.077142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:55.318084 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.318055 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:55.476495 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.476380 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls\") pod \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " Jul 06 17:09:55.476495 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.476414 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7shhg\" (UniqueName: \"kubernetes.io/projected/064812b2-c0e2-4071-8a08-ab1bb95c77bb-kube-api-access-7shhg\") pod \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " Jul 06 17:09:55.476734 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.476511 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064812b2-c0e2-4071-8a08-ab1bb95c77bb-openshift-service-ca\") pod \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\" (UID: \"064812b2-c0e2-4071-8a08-ab1bb95c77bb\") " Jul 06 17:09:55.476866 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.476835 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064812b2-c0e2-4071-8a08-ab1bb95c77bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "064812b2-c0e2-4071-8a08-ab1bb95c77bb" (UID: "064812b2-c0e2-4071-8a08-ab1bb95c77bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:09:55.478638 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.478598 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "064812b2-c0e2-4071-8a08-ab1bb95c77bb" (UID: "064812b2-c0e2-4071-8a08-ab1bb95c77bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:09:55.478638 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.478600 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064812b2-c0e2-4071-8a08-ab1bb95c77bb-kube-api-access-7shhg" (OuterVolumeSpecName: "kube-api-access-7shhg") pod "064812b2-c0e2-4071-8a08-ab1bb95c77bb" (UID: "064812b2-c0e2-4071-8a08-ab1bb95c77bb"). InnerVolumeSpecName "kube-api-access-7shhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:09:55.578227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.578196 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064812b2-c0e2-4071-8a08-ab1bb95c77bb-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:55.578227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.578224 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064812b2-c0e2-4071-8a08-ab1bb95c77bb-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:55.578227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.578235 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7shhg\" (UniqueName: \"kubernetes.io/projected/064812b2-c0e2-4071-8a08-ab1bb95c77bb-kube-api-access-7shhg\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:09:55.911340 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.911302 2574 generic.go:356] "Generic (PLEG): container finished" podID="064812b2-c0e2-4071-8a08-ab1bb95c77bb" containerID="98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c" exitCode=0 Jul 06 17:09:55.911340 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.911343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" event={"ID":"064812b2-c0e2-4071-8a08-ab1bb95c77bb","Type":"ContainerDied","Data":"98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c"} Jul 06 17:09:55.911839 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.911363 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" Jul 06 17:09:55.911839 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.911379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65" event={"ID":"064812b2-c0e2-4071-8a08-ab1bb95c77bb","Type":"ContainerDied","Data":"1fea554ec9997d715b115d9de787710e4793e3ffa29c7fe78c1a61508e437c2e"} Jul 06 17:09:55.911839 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.911399 2574 scope.go:117] "RemoveContainer" containerID="98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c" Jul 06 17:09:55.920038 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.920017 2574 scope.go:117] "RemoveContainer" containerID="98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c" Jul 06 17:09:55.920321 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:09:55.920294 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c\": container with ID starting with 98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c not found: ID does not exist" containerID="98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c" Jul 06 17:09:55.920426 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.920326 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c"} err="failed to get container status \"98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c\": rpc error: code = NotFound desc = could not find container \"98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c\": container with ID starting with 98a29067a717bb6f97e1b80be6e579b4684de33b23ae9886bc6a40b4e2c6d30c not found: ID does not exist" Jul 06 17:09:55.931143 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.931117 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65"] Jul 06 17:09:55.935509 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:55.935484 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-8b33da08-59565bc79-wft65"] Jul 06 17:09:56.207307 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:56.207279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:09:57.747473 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:09:57.747424 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064812b2-c0e2-4071-8a08-ab1bb95c77bb" path="/var/lib/kubelet/pods/064812b2-c0e2-4071-8a08-ab1bb95c77bb/volumes" Jul 06 17:10:01.140134 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:01.140099 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:01.247664 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:01.247625 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:01.616341 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:01.616305 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:01.718063 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:01.718013 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:01.820645 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:01.820612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:03.417131 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:03.417099 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:09.471781 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:09.471746 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:09.593260 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:09.593218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:09.686601 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:09.686568 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:09.790376 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:09.790326 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:10.327854 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:10.327816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:28.542386 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:28.542346 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:28.606969 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:28.606941 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:36.289469 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:36.289416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:42.324713 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:42.324676 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:42.424720 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:42.424685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:42.525249 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:42.525218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:10:58.441621 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:10:58.441588 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:05.410070 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:05.410035 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:05.512604 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:05.512575 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:05.611308 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:05.611279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:16.366205 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:16.366110 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:16.981955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:16.981925 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:22.928321 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:22.928288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:22.997956 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:22.997927 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:23.122858 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:23.122803 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:23.235848 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:23.235816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:24.031158 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:24.031119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:24.152315 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:24.152279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:24.248962 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:24.248932 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:25.922076 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:25.922036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:33.763404 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:33.763368 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:33.864240 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:33.864207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:34.455515 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:34.455447 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:41.045929 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.045885 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:41.122574 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.122536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:41.246183 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.246150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:41.535422 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.535386 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw"] Jul 06 17:11:41.535857 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.535840 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" containerName="istio-proxy" Jul 06 17:11:41.535857 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.535858 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" containerName="istio-proxy" Jul 06 17:11:41.535979 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.535882 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064812b2-c0e2-4071-8a08-ab1bb95c77bb" containerName="maas-api" Jul 06 17:11:41.535979 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.535891 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="064812b2-c0e2-4071-8a08-ab1bb95c77bb" containerName="maas-api" Jul 06 17:11:41.535979 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.535948 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" containerName="istio-proxy" Jul 06 17:11:41.535979 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.535959 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="064812b2-c0e2-4071-8a08-ab1bb95c77bb" containerName="maas-api" Jul 06 17:11:41.538674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.538651 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.541275 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.541252 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d72c5345-openshift-default-dockercfg-djbfh\"" Jul 06 17:11:41.563602 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.563567 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw"] Jul 06 17:11:41.598303 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.598261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:41.661216 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-socket\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.661216 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rtk\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-kube-api-access-r5rtk\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.661438 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-token\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.661438 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661264 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istiod-ca-cert\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.661438 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-certs\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.661438 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661359 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-podinfo\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.661438 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-credential-socket\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.661438 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661411 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-envoy\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.661647 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.661509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-data\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.704796 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.704765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:41.762388 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-data\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.762563 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-socket\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.762563 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rtk\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-kube-api-access-r5rtk\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.762563 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-token\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.762563 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istiod-ca-cert\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.762785 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-certs\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.762785 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-podinfo\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.762785 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-credential-socket\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.762785 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-envoy\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.763028 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762838 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-data\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.763028 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.762966 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-socket\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.763028 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.763005 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-certs\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.763191 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.763090 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-credential-socket\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.763374 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.763351 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istiod-ca-cert\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.764941 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.764923 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-envoy\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.765127 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.765108 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-podinfo\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.771016 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.770989 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-token\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.771110 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.771051 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rtk\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-kube-api-access-r5rtk\") pod \"e2e-mt-d72c5345-openshift-default-588974dd75-fklgw\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.848976 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.848889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:41.990316 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.990276 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw"] Jul 06 17:11:41.997740 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:11:41.997711 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccff89d_f1f8_4cde_b7fa_a66efcd781fd.slice/crio-8f04928c83d654d102e4849a4e20b79123700e3dcc8dadf5347de0f486df7849 WatchSource:0}: Error finding container 8f04928c83d654d102e4849a4e20b79123700e3dcc8dadf5347de0f486df7849: Status 404 returned error can't find the container with id 8f04928c83d654d102e4849a4e20b79123700e3dcc8dadf5347de0f486df7849 Jul 06 17:11:41.999794 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.999759 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:11:41.999908 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.999837 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:11:41.999908 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:41.999879 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:11:42.395612 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:42.395572 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" event={"ID":"cccff89d-f1f8-4cde-b7fa-a66efcd781fd","Type":"ContainerStarted","Data":"580fc43ee8ec30400a9a1940ed8eda5b56e7ce5a3d56f89df45a134bdd04b0ff"} Jul 06 17:11:42.396007 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:42.395631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" event={"ID":"cccff89d-f1f8-4cde-b7fa-a66efcd781fd","Type":"ContainerStarted","Data":"8f04928c83d654d102e4849a4e20b79123700e3dcc8dadf5347de0f486df7849"} Jul 06 17:11:42.418050 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:42.418002 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" podStartSLOduration=1.417984077 podStartE2EDuration="1.417984077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:11:42.416704492 +0000 UTC m=+1999.186596044" watchObservedRunningTime="2026-07-06 17:11:42.417984077 +0000 UTC m=+1999.187875617" Jul 06 17:11:42.849446 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:42.849404 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:43.854619 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:43.854588 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:44.404531 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.404503 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:44.405504 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.405472 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:44.532216 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.532161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:44.597039 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.597002 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p"] Jul 06 17:11:44.597426 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.597411 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" containerName="istio-proxy" Jul 06 17:11:44.597488 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.597428 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" containerName="istio-proxy" Jul 06 17:11:44.597532 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.597512 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="079a419a-7253-4198-b7dc-dd23bc58920b" containerName="istio-proxy" Jul 06 17:11:44.600623 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.600603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.603040 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.603018 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-d72c5345-serving-cert\"" Jul 06 17:11:44.611382 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.611353 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p"] Jul 06 17:11:44.613227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.613202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:44.684314 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.684215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.684314 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.684268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7hv\" (UniqueName: \"kubernetes.io/projected/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-kube-api-access-9c7hv\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.684567 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.684343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-openshift-service-ca\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.693740 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.693707 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:44.784987 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.784946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-openshift-service-ca\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.785196 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.785028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.785196 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.785063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7hv\" (UniqueName: \"kubernetes.io/projected/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-kube-api-access-9c7hv\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.785196 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:11:44.785180 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-d72c5345-serving-cert: secret "maas-api-e2e-mt-d72c5345-serving-cert" not found Jul 06 17:11:44.785384 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:11:44.785269 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls podName:bd69c00a-5ea9-437c-a3cc-3bf5892f1d06 nodeName:}" failed. No retries permitted until 2026-07-06 17:11:45.285246645 +0000 UTC m=+2002.055138186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls") pod "maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" (UID: "bd69c00a-5ea9-437c-a3cc-3bf5892f1d06") : secret "maas-api-e2e-mt-d72c5345-serving-cert" not found Jul 06 17:11:44.785640 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.785623 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-openshift-service-ca\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.799138 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.799112 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7hv\" (UniqueName: \"kubernetes.io/projected/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-kube-api-access-9c7hv\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:44.911634 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:44.911596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:45.196914 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:45.196874 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:45.290416 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:45.290378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:45.292960 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:45.292934 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls\") pod \"maas-api-e2e-mt-d72c5345-588d6f49b7-br86p\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:45.301067 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:45.301040 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:45.513323 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:45.513285 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:45.653538 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:45.653512 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p"] Jul 06 17:11:45.655682 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:11:45.655653 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd69c00a_5ea9_437c_a3cc_3bf5892f1d06.slice/crio-f62b6f1429687de811a78fe54312137218fd534695e61f7c55453be42b7d66e5 WatchSource:0}: Error finding container f62b6f1429687de811a78fe54312137218fd534695e61f7c55453be42b7d66e5: Status 404 returned error can't find the container with id f62b6f1429687de811a78fe54312137218fd534695e61f7c55453be42b7d66e5 Jul 06 17:11:46.415518 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:46.415394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" event={"ID":"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06","Type":"ContainerStarted","Data":"87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1"} Jul 06 17:11:46.415518 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:46.415485 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" event={"ID":"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06","Type":"ContainerStarted","Data":"f62b6f1429687de811a78fe54312137218fd534695e61f7c55453be42b7d66e5"} Jul 06 17:11:46.415962 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:46.415756 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:46.433032 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:46.432983 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" podStartSLOduration=1.945914084 podStartE2EDuration="2.432965678s" totalImagesPullingTime="487.051594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:11:46.432119075 +0000 UTC m=+2003.202010617" watchObservedRunningTime="2026-07-06 17:11:46.432965678 +0000 UTC m=+2003.202857219" Jul 06 17:11:48.977118 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:48.977083 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:49.621966 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:49.621931 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:49.720350 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:49.720316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:50.064256 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:50.064175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:52.425593 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:52.425563 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:56.224757 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:56.224723 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:56.323920 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:56.323884 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:56.799226 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:56.799194 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:57.067520 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.067425 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p"] Jul 06 17:11:57.067726 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.067695 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" podUID="bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" containerName="maas-api" containerID="cri-o://87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1" gracePeriod=30 Jul 06 17:11:57.135099 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.135063 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:57.235677 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.235601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:11:57.314285 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.314259 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:57.404303 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.404210 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls\") pod \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " Jul 06 17:11:57.404303 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.404248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-openshift-service-ca\") pod \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " Jul 06 17:11:57.404303 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.404266 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c7hv\" (UniqueName: \"kubernetes.io/projected/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-kube-api-access-9c7hv\") pod \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\" (UID: \"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06\") " Jul 06 17:11:57.404679 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.404651 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" (UID: "bd69c00a-5ea9-437c-a3cc-3bf5892f1d06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:11:57.406560 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.406532 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-kube-api-access-9c7hv" (OuterVolumeSpecName: "kube-api-access-9c7hv") pod "bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" (UID: "bd69c00a-5ea9-437c-a3cc-3bf5892f1d06"). InnerVolumeSpecName "kube-api-access-9c7hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:11:57.406872 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.406847 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" (UID: "bd69c00a-5ea9-437c-a3cc-3bf5892f1d06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:11:57.473542 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.473507 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" containerID="87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1" exitCode=0 Jul 06 17:11:57.473718 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.473548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" event={"ID":"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06","Type":"ContainerDied","Data":"87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1"} Jul 06 17:11:57.473718 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.473591 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" Jul 06 17:11:57.473718 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.473618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p" event={"ID":"bd69c00a-5ea9-437c-a3cc-3bf5892f1d06","Type":"ContainerDied","Data":"f62b6f1429687de811a78fe54312137218fd534695e61f7c55453be42b7d66e5"} Jul 06 17:11:57.473718 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.473640 2574 scope.go:117] "RemoveContainer" containerID="87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1" Jul 06 17:11:57.485788 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.485751 2574 scope.go:117] "RemoveContainer" containerID="87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1" Jul 06 17:11:57.486220 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:11:57.486189 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1\": container with ID starting with 87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1 not found: ID does not exist" containerID="87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1" Jul 06 17:11:57.486324 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.486237 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1"} err="failed to get container status \"87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1\": rpc error: code = NotFound desc = could not find container \"87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1\": container with ID starting with 87112968b611d68d48d93534a3a250d9a9a34c3e47fd6f60c6f12bcf10832da1 not found: ID does not exist" Jul 06 17:11:57.500836 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.500204 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p"] Jul 06 17:11:57.506025 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.505990 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:11:57.506025 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.506028 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:11:57.506243 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.506044 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7hv\" (UniqueName: \"kubernetes.io/projected/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06-kube-api-access-9c7hv\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:11:57.509011 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.508980 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-d72c5345-588d6f49b7-br86p"] Jul 06 17:11:57.748224 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:57.748192 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" path="/var/lib/kubelet/pods/bd69c00a-5ea9-437c-a3cc-3bf5892f1d06/volumes" Jul 06 17:11:58.481646 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:58.481565 2574 generic.go:356] "Generic (PLEG): container finished" podID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerID="580fc43ee8ec30400a9a1940ed8eda5b56e7ce5a3d56f89df45a134bdd04b0ff" exitCode=0 Jul 06 17:11:58.481646 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:58.481620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" event={"ID":"cccff89d-f1f8-4cde-b7fa-a66efcd781fd","Type":"ContainerDied","Data":"580fc43ee8ec30400a9a1940ed8eda5b56e7ce5a3d56f89df45a134bdd04b0ff"} Jul 06 17:11:58.482084 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:58.481951 2574 scope.go:117] "RemoveContainer" containerID="580fc43ee8ec30400a9a1940ed8eda5b56e7ce5a3d56f89df45a134bdd04b0ff" Jul 06 17:11:58.483072 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:58.483042 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:11:58.483147 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:58.483130 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:11:58.483200 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:58.483187 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:11:59.404730 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:59.404693 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:59.494124 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:59.494086 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" event={"ID":"cccff89d-f1f8-4cde-b7fa-a66efcd781fd","Type":"ContainerStarted","Data":"bd35c908f3c2be4fbe342c1ecb2bd9b4855169beb5be7a4fc04e03d414e27d20"} Jul 06 17:11:59.851340 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:59.851292 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:11:59.851575 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:59.851530 2574 patch_prober.go:28] interesting pod/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 06 17:11:59.851632 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:11:59.851594 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 06 17:12:00.850803 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:00.849717 2574 patch_prober.go:28] interesting pod/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 06 17:12:00.850803 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:00.849796 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 06 17:12:01.850263 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:01.849778 2574 patch_prober.go:28] interesting pod/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 06 17:12:01.850263 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:01.849850 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 06 17:12:02.849787 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:02.849734 2574 patch_prober.go:28] interesting pod/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 06 17:12:02.850324 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:02.849809 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 06 17:12:03.158693 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:03.158605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:03.295181 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:03.295149 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:03.403631 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:03.403595 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw"] Jul 06 17:12:03.403869 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:03.403841 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" containerID="cri-o://bd35c908f3c2be4fbe342c1ecb2bd9b4855169beb5be7a4fc04e03d414e27d20" gracePeriod=30 Jul 06 17:12:03.433134 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:03.433040 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:03.663509 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:03.663476 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:03.817533 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:03.817497 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:03.911061 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:03.911018 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:04.010477 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:04.010416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:08.540582 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.540547 2574 generic.go:356] "Generic (PLEG): container finished" podID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerID="bd35c908f3c2be4fbe342c1ecb2bd9b4855169beb5be7a4fc04e03d414e27d20" exitCode=0 Jul 06 17:12:08.541007 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.540606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" event={"ID":"cccff89d-f1f8-4cde-b7fa-a66efcd781fd","Type":"ContainerDied","Data":"bd35c908f3c2be4fbe342c1ecb2bd9b4855169beb5be7a4fc04e03d414e27d20"} Jul 06 17:12:08.541007 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.540655 2574 scope.go:117] "RemoveContainer" containerID="580fc43ee8ec30400a9a1940ed8eda5b56e7ce5a3d56f89df45a134bdd04b0ff" Jul 06 17:12:08.669742 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.669669 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:12:08.824285 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824231 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-token\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824285 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824283 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-envoy\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824578 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-socket\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824578 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824356 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5rtk\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-kube-api-access-r5rtk\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824578 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824396 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-podinfo\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824578 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824414 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istiod-ca-cert\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824578 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824496 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-certs\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824578 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824528 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-credential-socket\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824610 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-data\") pod \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\" (UID: \"cccff89d-f1f8-4cde-b7fa-a66efcd781fd\") " Jul 06 17:12:08.824874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824777 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:12:08.824874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824852 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:12:08.825049 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824990 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:08.825049 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.824998 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:12:08.825049 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.825010 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:08.825205 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.825047 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:12:08.825738 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.825708 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:12:08.826696 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.826673 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:12:08.826775 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.826706 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:12:08.826775 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.826715 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-kube-api-access-r5rtk" (OuterVolumeSpecName: "kube-api-access-r5rtk") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "kube-api-access-r5rtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:12:08.826950 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.826937 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "cccff89d-f1f8-4cde-b7fa-a66efcd781fd" (UID: "cccff89d-f1f8-4cde-b7fa-a66efcd781fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:12:08.926525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.926407 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:08.926525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.926440 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:08.926525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.926450 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:08.926525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.926481 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:08.926525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.926491 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:08.926525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.926498 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:08.926525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:08.926506 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5rtk\" (UniqueName: \"kubernetes.io/projected/cccff89d-f1f8-4cde-b7fa-a66efcd781fd-kube-api-access-r5rtk\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:12:09.545970 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:09.545936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" event={"ID":"cccff89d-f1f8-4cde-b7fa-a66efcd781fd","Type":"ContainerDied","Data":"8f04928c83d654d102e4849a4e20b79123700e3dcc8dadf5347de0f486df7849"} Jul 06 17:12:09.545970 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:09.545953 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw" Jul 06 17:12:09.546524 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:09.545994 2574 scope.go:117] "RemoveContainer" containerID="bd35c908f3c2be4fbe342c1ecb2bd9b4855169beb5be7a4fc04e03d414e27d20" Jul 06 17:12:09.571286 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:09.571254 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw"] Jul 06 17:12:09.575659 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:09.575634 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d72c5345-openshift-default-588974dd75-fklgw"] Jul 06 17:12:09.748274 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:09.748236 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" path="/var/lib/kubelet/pods/cccff89d-f1f8-4cde-b7fa-a66efcd781fd/volumes" Jul 06 17:12:10.196309 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:10.196269 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:10.302183 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:10.302151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:10.393132 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:10.393084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:10.492118 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:10.492088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:16.810688 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:16.810648 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:16.887212 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:16.887183 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:23.721416 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.721382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:23.999471 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.999425 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9"] Jul 06 17:12:23.999808 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.999795 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" Jul 06 17:12:23.999854 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.999811 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" Jul 06 17:12:23.999854 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.999825 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" containerName="maas-api" Jul 06 17:12:23.999854 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.999830 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" containerName="maas-api" Jul 06 17:12:23.999953 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.999882 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd69c00a-5ea9-437c-a3cc-3bf5892f1d06" containerName="maas-api" Jul 06 17:12:23.999953 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.999892 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" Jul 06 17:12:23.999953 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:23.999902 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" Jul 06 17:12:24.004509 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.004489 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.006967 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.006943 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8bfb9318-openshift-default-dockercfg-rh54d\"" Jul 06 17:12:24.017124 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.017097 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9"] Jul 06 17:12:24.023162 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.023140 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:24.061018 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.060986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-socket\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.061018 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.061023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istiod-ca-cert\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.061229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.061048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-data\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.061229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.061084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqxz\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-kube-api-access-grqxz\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.061229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.061115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-certs\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.061229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.061137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-podinfo\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.061229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.061203 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-envoy\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.061396 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.061248 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-credential-socket\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.061396 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.061265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-token\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.120070 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.120036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:24.161757 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.161716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-podinfo\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.161955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.161791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-envoy\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.161955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.161842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-credential-socket\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.161955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.161865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-token\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.161955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.161929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-socket\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162177 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.161965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istiod-ca-cert\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162177 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.162002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-data\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162177 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.162028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqxz\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-kube-api-access-grqxz\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162177 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.162059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-certs\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162484 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.162337 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-socket\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162484 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.162420 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-credential-socket\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162612 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.162508 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-data\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162683 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.162664 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istiod-ca-cert\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.162788 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.162768 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-certs\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.164260 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.164240 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-envoy\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.164385 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.164370 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-podinfo\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.170764 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.170743 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-token\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.170944 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.170924 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqxz\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-kube-api-access-grqxz\") pod \"e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.315193 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.315108 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:24.450931 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.450755 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9"] Jul 06 17:12:24.453287 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:12:24.453256 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b6469d_64bb_46e4_9e86_aaa0f3363a73.slice/crio-60f4a212d0db0bc16323a182d9f8c55552311f1d28ff88fccbc03d834d8fadd1 WatchSource:0}: Error finding container 60f4a212d0db0bc16323a182d9f8c55552311f1d28ff88fccbc03d834d8fadd1: Status 404 returned error can't find the container with id 60f4a212d0db0bc16323a182d9f8c55552311f1d28ff88fccbc03d834d8fadd1 Jul 06 17:12:24.455433 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.455402 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:12:24.455541 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.455488 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:12:24.455541 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.455516 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:12:24.619044 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.618940 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" event={"ID":"b5b6469d-64bb-46e4-9e86-aaa0f3363a73","Type":"ContainerStarted","Data":"a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096"} Jul 06 17:12:24.619044 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.619029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" event={"ID":"b5b6469d-64bb-46e4-9e86-aaa0f3363a73","Type":"ContainerStarted","Data":"60f4a212d0db0bc16323a182d9f8c55552311f1d28ff88fccbc03d834d8fadd1"} Jul 06 17:12:24.644212 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:24.644153 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" podStartSLOduration=1.644131016 podStartE2EDuration="1.644131016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:12:24.641247336 +0000 UTC m=+2041.411138878" watchObservedRunningTime="2026-07-06 17:12:24.644131016 +0000 UTC m=+2041.414022558" Jul 06 17:12:25.316337 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:25.316298 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:25.317702 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:25.317679 2574 patch_prober.go:28] interesting pod/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 06 17:12:25.317765 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:25.317730 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 06 17:12:26.320347 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.320315 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:26.630059 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.629977 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:26.631034 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.630995 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:26.683570 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.683536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:26.800439 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.800406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:26.808298 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.808271 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v"] Jul 06 17:12:26.808724 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.808707 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" Jul 06 17:12:26.808809 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.808726 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccff89d-f1f8-4cde-b7fa-a66efcd781fd" containerName="istio-proxy" Jul 06 17:12:26.811688 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.811667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:26.814441 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.814421 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-8bfb9318-serving-cert\"" Jul 06 17:12:26.829359 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.829330 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v"] Jul 06 17:12:26.889674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.889573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecb43bd-9873-49c9-bf93-c4a34082f300-maas-api-tls\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:26.889845 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.889693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecb43bd-9873-49c9-bf93-c4a34082f300-openshift-service-ca\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:26.889845 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.889744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqzm\" (UniqueName: \"kubernetes.io/projected/5ecb43bd-9873-49c9-bf93-c4a34082f300-kube-api-access-xpqzm\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:26.895094 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.895063 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:26.991115 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.991080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecb43bd-9873-49c9-bf93-c4a34082f300-openshift-service-ca\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:26.991115 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.991114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqzm\" (UniqueName: \"kubernetes.io/projected/5ecb43bd-9873-49c9-bf93-c4a34082f300-kube-api-access-xpqzm\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:26.991332 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.991198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecb43bd-9873-49c9-bf93-c4a34082f300-maas-api-tls\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:26.991751 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.991730 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecb43bd-9873-49c9-bf93-c4a34082f300-openshift-service-ca\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:26.993641 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:26.993620 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecb43bd-9873-49c9-bf93-c4a34082f300-maas-api-tls\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:27.004751 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:27.004729 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqzm\" (UniqueName: \"kubernetes.io/projected/5ecb43bd-9873-49c9-bf93-c4a34082f300-kube-api-access-xpqzm\") pod \"maas-api-e2e-mt-8bfb9318-68889595c-p2j4v\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:27.017222 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:27.017194 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:27.122133 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:27.122094 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:27.265791 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:27.265764 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v"] Jul 06 17:12:27.268011 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:12:27.267981 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecb43bd_9873_49c9_bf93_c4a34082f300.slice/crio-cf9f1ac582f5413ef65984bad4feb5034470d7e89678d95c5b3878706553bcd8 WatchSource:0}: Error finding container cf9f1ac582f5413ef65984bad4feb5034470d7e89678d95c5b3878706553bcd8: Status 404 returned error can't find the container with id cf9f1ac582f5413ef65984bad4feb5034470d7e89678d95c5b3878706553bcd8 Jul 06 17:12:27.636091 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:27.636054 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" event={"ID":"5ecb43bd-9873-49c9-bf93-c4a34082f300","Type":"ContainerStarted","Data":"cf9f1ac582f5413ef65984bad4feb5034470d7e89678d95c5b3878706553bcd8"} Jul 06 17:12:27.802276 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:27.802247 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:27.901325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:27.901241 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:28.641828 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:28.641789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" event={"ID":"5ecb43bd-9873-49c9-bf93-c4a34082f300","Type":"ContainerStarted","Data":"ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b"} Jul 06 17:12:28.661377 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:28.661321 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" podStartSLOduration=2.167367502 podStartE2EDuration="2.661306048s" totalImagesPullingTime="493.938546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:12:28.659768338 +0000 UTC m=+2045.429659905" watchObservedRunningTime="2026-07-06 17:12:28.661306048 +0000 UTC m=+2045.431197589" Jul 06 17:12:29.646515 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:29.646482 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:31.155043 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:31.155012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:35.656470 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:35.656430 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:12:40.332311 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:40.332269 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:40.432658 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:40.432624 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:40.701360 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:40.701272 2574 generic.go:356] "Generic (PLEG): container finished" podID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerID="a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096" exitCode=0 Jul 06 17:12:40.701548 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:40.701351 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" event={"ID":"b5b6469d-64bb-46e4-9e86-aaa0f3363a73","Type":"ContainerDied","Data":"a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096"} Jul 06 17:12:40.701833 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:40.701819 2574 scope.go:117] "RemoveContainer" containerID="a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096" Jul 06 17:12:40.702783 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:40.702760 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:12:40.702837 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:40.702822 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:12:40.702872 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:40.702851 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:12:41.630475 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:41.630411 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:41.715157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:41.715123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" event={"ID":"b5b6469d-64bb-46e4-9e86-aaa0f3363a73","Type":"ContainerStarted","Data":"d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859"} Jul 06 17:12:42.315656 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:42.315616 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:42.316664 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:42.316638 2574 patch_prober.go:28] interesting pod/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 06 17:12:42.316790 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:42.316688 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 06 17:12:43.315888 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:43.315851 2574 patch_prober.go:28] interesting pod/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 06 17:12:43.316291 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:43.315917 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 06 17:12:44.315716 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:44.315680 2574 patch_prober.go:28] interesting pod/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 06 17:12:44.315904 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:44.315746 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 06 17:12:45.315532 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:45.315432 2574 patch_prober.go:28] interesting pod/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 06 17:12:45.315532 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:45.315506 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 06 17:12:46.319733 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:46.319702 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:46.320125 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:46.319968 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:46.320692 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:46.320671 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:12:46.545748 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:46.545712 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:48.548069 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:48.548028 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:12:48.648022 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:12:48.647991 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:13:13.841931 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:13.841891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:13:14.156931 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.156841 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v"] Jul 06 17:13:14.157126 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.157099 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" podUID="5ecb43bd-9873-49c9-bf93-c4a34082f300" containerName="maas-api" containerID="cri-o://ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b" gracePeriod=30 Jul 06 17:13:14.182343 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.182314 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:13:14.406911 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.406880 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:13:14.448267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.448180 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecb43bd-9873-49c9-bf93-c4a34082f300-openshift-service-ca\") pod \"5ecb43bd-9873-49c9-bf93-c4a34082f300\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " Jul 06 17:13:14.448267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.448244 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecb43bd-9873-49c9-bf93-c4a34082f300-maas-api-tls\") pod \"5ecb43bd-9873-49c9-bf93-c4a34082f300\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " Jul 06 17:13:14.448482 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.448273 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpqzm\" (UniqueName: \"kubernetes.io/projected/5ecb43bd-9873-49c9-bf93-c4a34082f300-kube-api-access-xpqzm\") pod \"5ecb43bd-9873-49c9-bf93-c4a34082f300\" (UID: \"5ecb43bd-9873-49c9-bf93-c4a34082f300\") " Jul 06 17:13:14.448610 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.448574 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecb43bd-9873-49c9-bf93-c4a34082f300-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ecb43bd-9873-49c9-bf93-c4a34082f300" (UID: "5ecb43bd-9873-49c9-bf93-c4a34082f300"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:13:14.448740 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.448652 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecb43bd-9873-49c9-bf93-c4a34082f300-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:13:14.450345 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.450321 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecb43bd-9873-49c9-bf93-c4a34082f300-kube-api-access-xpqzm" (OuterVolumeSpecName: "kube-api-access-xpqzm") pod "5ecb43bd-9873-49c9-bf93-c4a34082f300" (UID: "5ecb43bd-9873-49c9-bf93-c4a34082f300"). InnerVolumeSpecName "kube-api-access-xpqzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:13:14.450431 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.450341 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecb43bd-9873-49c9-bf93-c4a34082f300-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ecb43bd-9873-49c9-bf93-c4a34082f300" (UID: "5ecb43bd-9873-49c9-bf93-c4a34082f300"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:13:14.549885 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.549845 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecb43bd-9873-49c9-bf93-c4a34082f300-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:13:14.549885 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.549878 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqzm\" (UniqueName: \"kubernetes.io/projected/5ecb43bd-9873-49c9-bf93-c4a34082f300-kube-api-access-xpqzm\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:13:14.879385 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.879346 2574 generic.go:356] "Generic (PLEG): container finished" podID="5ecb43bd-9873-49c9-bf93-c4a34082f300" containerID="ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b" exitCode=0 Jul 06 17:13:14.879801 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.879396 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" event={"ID":"5ecb43bd-9873-49c9-bf93-c4a34082f300","Type":"ContainerDied","Data":"ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b"} Jul 06 17:13:14.879801 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.879405 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" Jul 06 17:13:14.879801 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.879500 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v" event={"ID":"5ecb43bd-9873-49c9-bf93-c4a34082f300","Type":"ContainerDied","Data":"cf9f1ac582f5413ef65984bad4feb5034470d7e89678d95c5b3878706553bcd8"} Jul 06 17:13:14.879801 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.879530 2574 scope.go:117] "RemoveContainer" containerID="ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b" Jul 06 17:13:14.888678 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.888661 2574 scope.go:117] "RemoveContainer" containerID="ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b" Jul 06 17:13:14.888915 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:13:14.888900 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b\": container with ID starting with ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b not found: ID does not exist" containerID="ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b" Jul 06 17:13:14.888968 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.888922 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b"} err="failed to get container status \"ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b\": rpc error: code = NotFound desc = could not find container \"ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b\": container with ID starting with ec7ca68ecac47806ab5abdec79b4c408154b1577cf03e489798ea0734d09e63b not found: ID does not exist" Jul 06 17:13:14.905112 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.905084 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v"] Jul 06 17:13:14.908897 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:14.908871 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-8bfb9318-68889595c-p2j4v"] Jul 06 17:13:15.748759 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:15.748726 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecb43bd-9873-49c9-bf93-c4a34082f300" path="/var/lib/kubelet/pods/5ecb43bd-9873-49c9-bf93-c4a34082f300/volumes" Jul 06 17:13:20.035663 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:20.035627 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:13:25.110580 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:25.110546 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:13:53.155283 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:13:53.155250 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:01.274512 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:01.274474 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:02.919954 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:02.919902 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:08.966172 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:08.966133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:09.039714 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:09.039680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:09.266683 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:09.266648 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9"] Jul 06 17:14:09.266873 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:09.266836 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" containerID="cri-o://d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859" gracePeriod=30 Jul 06 17:14:09.275258 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:09.275221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:09.527577 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:09.527490 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:10.838183 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:10.838144 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:10.928607 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:10.928569 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:11.958228 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:11.958191 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:12.087355 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:12.087322 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:12.180853 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:12.180811 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:13.724625 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:13.724596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:14.525803 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.525777 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:14:14.613953 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.613850 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-socket\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.613953 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.613888 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-credential-socket\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.613953 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.613938 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-certs\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.614248 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.613974 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-data\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.614248 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614013 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-token\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.614248 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614040 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grqxz\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-kube-api-access-grqxz\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.614248 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614105 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-podinfo\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.614248 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614134 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istiod-ca-cert\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.614248 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614151 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:14:14.614248 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614180 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-envoy\") pod \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\" (UID: \"b5b6469d-64bb-46e4-9e86-aaa0f3363a73\") " Jul 06 17:14:14.614613 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614313 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:14:14.614613 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614346 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:14:14.614613 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614597 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:14.614789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614620 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:14.614789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614635 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:14.614789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.614650 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:14:14.615076 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.615039 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:14:14.616664 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.616633 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:14:14.616664 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.616644 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:14:14.616664 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.616659 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-kube-api-access-grqxz" (OuterVolumeSpecName: "kube-api-access-grqxz") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "kube-api-access-grqxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:14:14.616899 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.616692 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5b6469d-64bb-46e4-9e86-aaa0f3363a73" (UID: "b5b6469d-64bb-46e4-9e86-aaa0f3363a73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:14:14.715347 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.715309 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:14.715347 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.715341 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:14.715347 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.715352 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grqxz\" (UniqueName: \"kubernetes.io/projected/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-kube-api-access-grqxz\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:14.715627 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.715367 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:14.715627 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.715377 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:14.715627 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:14.715387 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6469d-64bb-46e4-9e86-aaa0f3363a73-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:14:15.159270 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.159240 2574 generic.go:356] "Generic (PLEG): container finished" podID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerID="d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859" exitCode=0 Jul 06 17:14:15.159732 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.159289 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" Jul 06 17:14:15.159732 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.159329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" event={"ID":"b5b6469d-64bb-46e4-9e86-aaa0f3363a73","Type":"ContainerDied","Data":"d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859"} Jul 06 17:14:15.159732 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.159385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9" event={"ID":"b5b6469d-64bb-46e4-9e86-aaa0f3363a73","Type":"ContainerDied","Data":"60f4a212d0db0bc16323a182d9f8c55552311f1d28ff88fccbc03d834d8fadd1"} Jul 06 17:14:15.159732 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.159410 2574 scope.go:117] "RemoveContainer" containerID="d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859" Jul 06 17:14:15.168874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.168855 2574 scope.go:117] "RemoveContainer" containerID="a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096" Jul 06 17:14:15.176923 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.176905 2574 scope.go:117] "RemoveContainer" containerID="d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859" Jul 06 17:14:15.177180 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:14:15.177161 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859\": container with ID starting with d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859 not found: ID does not exist" containerID="d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859" Jul 06 17:14:15.177229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.177189 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859"} err="failed to get container status \"d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859\": rpc error: code = NotFound desc = could not find container \"d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859\": container with ID starting with d94d48887e1d0701e7ae0f5029c0a26b0bd4c3b335e64d9561025e947adf1859 not found: ID does not exist" Jul 06 17:14:15.177229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.177206 2574 scope.go:117] "RemoveContainer" containerID="a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096" Jul 06 17:14:15.177472 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:14:15.177441 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096\": container with ID starting with a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096 not found: ID does not exist" containerID="a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096" Jul 06 17:14:15.177538 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.177483 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096"} err="failed to get container status \"a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096\": rpc error: code = NotFound desc = could not find container \"a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096\": container with ID starting with a298942f2fe3d54285323fff159781a6403ae074227984639eb57111d3781096 not found: ID does not exist" Jul 06 17:14:15.184224 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.184201 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9"] Jul 06 17:14:15.188331 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.188306 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8bfb9318-openshift-default-5884775ff-vwnl9"] Jul 06 17:14:15.748129 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:15.748095 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" path="/var/lib/kubelet/pods/b5b6469d-64bb-46e4-9e86-aaa0f3363a73/volumes" Jul 06 17:14:19.878563 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:19.878521 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:19.978295 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:19.978263 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:20.636788 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:20.636749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:27.004767 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:27.004735 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:27.076998 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:27.076966 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:27.176948 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:27.176909 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:27.275631 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:27.275548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:37.779410 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:37.779370 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:38.140944 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:38.140832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:14:59.272499 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:14:59.272434 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:00.148597 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.148555 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722635-md567"] Jul 06 17:15:00.149010 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.148992 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" Jul 06 17:15:00.149100 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.149013 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" Jul 06 17:15:00.149100 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.149052 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecb43bd-9873-49c9-bf93-c4a34082f300" containerName="maas-api" Jul 06 17:15:00.149100 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.149063 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecb43bd-9873-49c9-bf93-c4a34082f300" containerName="maas-api" Jul 06 17:15:00.149258 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.149141 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecb43bd-9873-49c9-bf93-c4a34082f300" containerName="maas-api" Jul 06 17:15:00.149258 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.149155 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" Jul 06 17:15:00.149258 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.149165 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" Jul 06 17:15:00.152170 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.152139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722635-md567" Jul 06 17:15:00.161612 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.161571 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722635-md567"] Jul 06 17:15:00.235244 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.235199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8hzq\" (UniqueName: \"kubernetes.io/projected/c7b17d09-4c54-478c-9c89-805c025d9230-kube-api-access-s8hzq\") pod \"maas-api-key-cleanup-29722635-md567\" (UID: \"c7b17d09-4c54-478c-9c89-805c025d9230\") " pod="opendatahub/maas-api-key-cleanup-29722635-md567" Jul 06 17:15:00.336279 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.336241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8hzq\" (UniqueName: \"kubernetes.io/projected/c7b17d09-4c54-478c-9c89-805c025d9230-kube-api-access-s8hzq\") pod \"maas-api-key-cleanup-29722635-md567\" (UID: \"c7b17d09-4c54-478c-9c89-805c025d9230\") " pod="opendatahub/maas-api-key-cleanup-29722635-md567" Jul 06 17:15:00.345388 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.345361 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8hzq\" (UniqueName: \"kubernetes.io/projected/c7b17d09-4c54-478c-9c89-805c025d9230-kube-api-access-s8hzq\") pod \"maas-api-key-cleanup-29722635-md567\" (UID: \"c7b17d09-4c54-478c-9c89-805c025d9230\") " pod="opendatahub/maas-api-key-cleanup-29722635-md567" Jul 06 17:15:00.464801 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.464711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722635-md567" Jul 06 17:15:00.609651 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:00.609625 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722635-md567"] Jul 06 17:15:00.612514 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:15:00.612481 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b17d09_4c54_478c_9c89_805c025d9230.slice/crio-e4079f781a74a1eec189952a1b2e9bfe9c4a0cfe7f6589ce9da1117b8e4b4a3f WatchSource:0}: Error finding container e4079f781a74a1eec189952a1b2e9bfe9c4a0cfe7f6589ce9da1117b8e4b4a3f: Status 404 returned error can't find the container with id e4079f781a74a1eec189952a1b2e9bfe9c4a0cfe7f6589ce9da1117b8e4b4a3f Jul 06 17:15:01.374129 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:01.374087 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722635-md567" event={"ID":"c7b17d09-4c54-478c-9c89-805c025d9230","Type":"ContainerStarted","Data":"0518d03f0a9a5b41fc53b95d60e1556d803300ab2085913b8e69228f5fc24a5a"} Jul 06 17:15:01.374602 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:01.374144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722635-md567" event={"ID":"c7b17d09-4c54-478c-9c89-805c025d9230","Type":"ContainerStarted","Data":"e4079f781a74a1eec189952a1b2e9bfe9c4a0cfe7f6589ce9da1117b8e4b4a3f"} Jul 06 17:15:01.400171 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:01.400118 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722635-md567" podStartSLOduration=1.400102129 podStartE2EDuration="1.400102129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:15:01.398791846 +0000 UTC m=+2198.168683402" watchObservedRunningTime="2026-07-06 17:15:01.400102129 +0000 UTC m=+2198.169993704" Jul 06 17:15:02.380216 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:02.380178 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7b17d09-4c54-478c-9c89-805c025d9230" containerID="0518d03f0a9a5b41fc53b95d60e1556d803300ab2085913b8e69228f5fc24a5a" exitCode=0 Jul 06 17:15:02.380660 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:02.380264 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722635-md567" event={"ID":"c7b17d09-4c54-478c-9c89-805c025d9230","Type":"ContainerDied","Data":"0518d03f0a9a5b41fc53b95d60e1556d803300ab2085913b8e69228f5fc24a5a"} Jul 06 17:15:03.523881 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:03.523848 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722635-md567" Jul 06 17:15:03.668997 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:03.668901 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8hzq\" (UniqueName: \"kubernetes.io/projected/c7b17d09-4c54-478c-9c89-805c025d9230-kube-api-access-s8hzq\") pod \"c7b17d09-4c54-478c-9c89-805c025d9230\" (UID: \"c7b17d09-4c54-478c-9c89-805c025d9230\") " Jul 06 17:15:03.671115 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:03.671076 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b17d09-4c54-478c-9c89-805c025d9230-kube-api-access-s8hzq" (OuterVolumeSpecName: "kube-api-access-s8hzq") pod "c7b17d09-4c54-478c-9c89-805c025d9230" (UID: "c7b17d09-4c54-478c-9c89-805c025d9230"). InnerVolumeSpecName "kube-api-access-s8hzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:15:03.770447 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:03.770408 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8hzq\" (UniqueName: \"kubernetes.io/projected/c7b17d09-4c54-478c-9c89-805c025d9230-kube-api-access-s8hzq\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:15:04.390271 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:04.390234 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722635-md567" Jul 06 17:15:04.390271 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:04.390263 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722635-md567" event={"ID":"c7b17d09-4c54-478c-9c89-805c025d9230","Type":"ContainerDied","Data":"e4079f781a74a1eec189952a1b2e9bfe9c4a0cfe7f6589ce9da1117b8e4b4a3f"} Jul 06 17:15:04.390531 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:04.390316 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4079f781a74a1eec189952a1b2e9bfe9c4a0cfe7f6589ce9da1117b8e4b4a3f" Jul 06 17:15:05.658411 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:05.658365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:05.731695 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:05.731660 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:06.027186 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.027151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:06.309429 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.309334 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs"] Jul 06 17:15:06.309811 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.309793 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" Jul 06 17:15:06.309899 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.309813 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b6469d-64bb-46e4-9e86-aaa0f3363a73" containerName="istio-proxy" Jul 06 17:15:06.309899 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.309827 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b17d09-4c54-478c-9c89-805c025d9230" containerName="cleanup" Jul 06 17:15:06.309899 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.309835 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b17d09-4c54-478c-9c89-805c025d9230" containerName="cleanup" Jul 06 17:15:06.310048 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.309930 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b17d09-4c54-478c-9c89-805c025d9230" containerName="cleanup" Jul 06 17:15:06.312933 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.312911 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.315756 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.315733 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cdb744-openshift-default-dockercfg-th4cx\"" Jul 06 17:15:06.324654 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.324630 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs"] Jul 06 17:15:06.334928 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.334890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:06.496948 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.496910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-podinfo\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.497124 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.496960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-credential-socket\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.497124 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.497007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-certs\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.497124 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.497039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fksj\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-kube-api-access-8fksj\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.497124 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.497069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-socket\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.497124 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.497094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-envoy\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.497297 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.497126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-data\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.497297 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.497187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-token\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.497297 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.497204 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istiod-ca-cert\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.510929 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.510888 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:06.598196 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-certs\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598196 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fksj\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-kube-api-access-8fksj\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598196 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-socket\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598196 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-envoy\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598567 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-data\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598567 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-token\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598567 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istiod-ca-cert\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598567 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-podinfo\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598567 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-credential-socket\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598796 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598637 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-socket\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598796 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598678 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-certs\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598883 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598810 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-credential-socket\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.598883 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.598821 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-data\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.599022 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.599002 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istiod-ca-cert\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.600626 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.600605 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-envoy\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.600822 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.600805 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-podinfo\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.606262 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.606236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-token\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.606262 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.606252 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fksj\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-kube-api-access-8fksj\") pod \"e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.624270 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.624236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:06.765307 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.765280 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs"] Jul 06 17:15:06.767862 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:15:06.767833 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd175a1de_444f_4cf4_8516_d0b05d9e32bd.slice/crio-d527b1adcccd518f4a797a55d219fae7eb610f77199f5fb9ad855b4d59e68845 WatchSource:0}: Error finding container d527b1adcccd518f4a797a55d219fae7eb610f77199f5fb9ad855b4d59e68845: Status 404 returned error can't find the container with id d527b1adcccd518f4a797a55d219fae7eb610f77199f5fb9ad855b4d59e68845 Jul 06 17:15:06.770095 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.770065 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:06.770168 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.770132 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:06.770168 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:06.770159 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:07.406699 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:07.406663 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" event={"ID":"d175a1de-444f-4cf4-8516-d0b05d9e32bd","Type":"ContainerStarted","Data":"786fed8f0ee235f51d75bbbcf31b7ac3612f365182f364e456f8967ca3323280"} Jul 06 17:15:07.406881 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:07.406718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" event={"ID":"d175a1de-444f-4cf4-8516-d0b05d9e32bd","Type":"ContainerStarted","Data":"d527b1adcccd518f4a797a55d219fae7eb610f77199f5fb9ad855b4d59e68845"} Jul 06 17:15:07.431654 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:07.431596 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" podStartSLOduration=1.431578399 podStartE2EDuration="1.431578399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:15:07.429260277 +0000 UTC m=+2204.199151819" watchObservedRunningTime="2026-07-06 17:15:07.431578399 +0000 UTC m=+2204.201470012" Jul 06 17:15:07.625371 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:07.625326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:07.630377 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:07.630351 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:08.411074 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:08.411043 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:08.411920 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:08.411901 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:09.506069 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.506035 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:09.610372 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.610317 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:09.677899 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.677860 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7"] Jul 06 17:15:09.681808 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.681787 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.684477 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.684419 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-cdb744-serving-cert\"" Jul 06 17:15:09.695105 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.695075 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7"] Jul 06 17:15:09.726291 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.726258 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:09.826479 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.826377 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:09.826634 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.826613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rq2f\" (UniqueName: \"kubernetes.io/projected/695307e6-64fe-4022-8fb2-379478f49065-kube-api-access-5rq2f\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.826680 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.826658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695307e6-64fe-4022-8fb2-379478f49065-maas-api-tls\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.826781 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.826770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695307e6-64fe-4022-8fb2-379478f49065-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.927271 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.927231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rq2f\" (UniqueName: \"kubernetes.io/projected/695307e6-64fe-4022-8fb2-379478f49065-kube-api-access-5rq2f\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.927271 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.927277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695307e6-64fe-4022-8fb2-379478f49065-maas-api-tls\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.927515 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.927345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695307e6-64fe-4022-8fb2-379478f49065-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.928126 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.928106 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695307e6-64fe-4022-8fb2-379478f49065-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.929829 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.929803 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695307e6-64fe-4022-8fb2-379478f49065-maas-api-tls\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.937133 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.937113 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rq2f\" (UniqueName: \"kubernetes.io/projected/695307e6-64fe-4022-8fb2-379478f49065-kube-api-access-5rq2f\") pod \"maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:09.997960 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:09.997924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:10.140049 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:10.140022 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7"] Jul 06 17:15:10.142155 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:15:10.142124 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695307e6_64fe_4022_8fb2_379478f49065.slice/crio-8a4eba961815fed53d3c6d882598b8bf78187f384e5c34176d48abaeb426f7ad WatchSource:0}: Error finding container 8a4eba961815fed53d3c6d882598b8bf78187f384e5c34176d48abaeb426f7ad: Status 404 returned error can't find the container with id 8a4eba961815fed53d3c6d882598b8bf78187f384e5c34176d48abaeb426f7ad Jul 06 17:15:10.143803 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:10.143782 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 17:15:10.155108 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:10.155084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:10.255350 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:10.255310 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:10.421662 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:10.421563 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" event={"ID":"695307e6-64fe-4022-8fb2-379478f49065","Type":"ContainerStarted","Data":"8a4eba961815fed53d3c6d882598b8bf78187f384e5c34176d48abaeb426f7ad"} Jul 06 17:15:11.427129 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:11.427092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" event={"ID":"695307e6-64fe-4022-8fb2-379478f49065","Type":"ContainerStarted","Data":"1a4307e3496238127e93a3640813e8b757a93473df73136e97458db734093323"} Jul 06 17:15:11.427556 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:11.427194 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:11.446898 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:11.446846 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" podStartSLOduration=1.868920096 podStartE2EDuration="2.446830847s" totalImagesPullingTime="577.910751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:15:11.445770673 +0000 UTC m=+2208.215662215" watchObservedRunningTime="2026-07-06 17:15:11.446830847 +0000 UTC m=+2208.216722389" Jul 06 17:15:14.010572 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.010539 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:14.401414 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.401304 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:14.674279 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.674190 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs"] Jul 06 17:15:14.678118 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.678097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.680697 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.680678 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ebaa5d-openshift-default-dockercfg-mp687\"" Jul 06 17:15:14.692392 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.692359 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs"] Jul 06 17:15:14.715165 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.715133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:14.771532 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771497 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-envoy\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.771712 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81158c40-bb7f-4785-841f-9a736ed31e2a-istiod-ca-cert\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.771712 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2ccq\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-kube-api-access-r2ccq\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.771712 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-data\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.771846 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-token\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.771846 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-credential-socket\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.771846 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-socket\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.771960 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-certs\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.771960 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.771894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-podinfo\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873016 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.872983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-token\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873016 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-credential-socket\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-socket\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-certs\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-podinfo\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-envoy\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81158c40-bb7f-4785-841f-9a736ed31e2a-istiod-ca-cert\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2ccq\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-kube-api-access-r2ccq\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873267 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-data\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873444 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-credential-socket\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873674 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873539 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-socket\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873784 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873670 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-data\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873883 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873865 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-certs\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.873997 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.873979 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81158c40-bb7f-4785-841f-9a736ed31e2a-istiod-ca-cert\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.875439 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.875418 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-envoy\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.875558 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.875542 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-podinfo\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.881183 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.881157 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-token\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.881284 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.881219 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2ccq\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-kube-api-access-r2ccq\") pod \"e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:14.947193 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.947102 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:14.988860 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:14.988819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:15.131273 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:15.131248 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs"] Jul 06 17:15:15.133126 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:15:15.133101 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81158c40_bb7f_4785_841f_9a736ed31e2a.slice/crio-cf3686ec428736be1291346801536ecac1e360050047622e1663c65a5eae8570 WatchSource:0}: Error finding container cf3686ec428736be1291346801536ecac1e360050047622e1663c65a5eae8570: Status 404 returned error can't find the container with id cf3686ec428736be1291346801536ecac1e360050047622e1663c65a5eae8570 Jul 06 17:15:15.135300 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:15.135270 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:15.135402 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:15.135334 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:15.135402 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:15.135363 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:15.448510 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:15.448472 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" event={"ID":"81158c40-bb7f-4785-841f-9a736ed31e2a","Type":"ContainerStarted","Data":"61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9"} Jul 06 17:15:15.448711 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:15.448537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" event={"ID":"81158c40-bb7f-4785-841f-9a736ed31e2a","Type":"ContainerStarted","Data":"cf3686ec428736be1291346801536ecac1e360050047622e1663c65a5eae8570"} Jul 06 17:15:15.471841 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:15.471791 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" podStartSLOduration=1.4717737309999999 podStartE2EDuration="1.471773731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:15:15.469904091 +0000 UTC m=+2212.239795643" watchObservedRunningTime="2026-07-06 17:15:15.471773731 +0000 UTC m=+2212.241665273" Jul 06 17:15:15.989877 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:15.989843 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:16.994439 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:16.994402 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:17.437428 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.437338 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:15:17.459891 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.459810 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:17.460747 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.460722 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:17.538404 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.538366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:17.616105 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.616077 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2"] Jul 06 17:15:17.619805 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.619770 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.622833 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.622808 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-ebaa5d-serving-cert\"" Jul 06 17:15:17.628265 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.628238 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2"] Jul 06 17:15:17.631665 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.631624 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:17.705104 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.705000 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ecf64-8e04-4c31-8128-7af4855b2368-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.705104 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.705051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxn2g\" (UniqueName: \"kubernetes.io/projected/105ecf64-8e04-4c31-8128-7af4855b2368-kube-api-access-sxn2g\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.705334 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.705176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.717068 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.717004 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:17.806322 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.806286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.806571 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.806359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ecf64-8e04-4c31-8128-7af4855b2368-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.806571 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.806399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxn2g\" (UniqueName: \"kubernetes.io/projected/105ecf64-8e04-4c31-8128-7af4855b2368-kube-api-access-sxn2g\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.806571 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:15:17.806440 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-ebaa5d-serving-cert: secret "maas-api-e2e-inf-b-ebaa5d-serving-cert" not found Jul 06 17:15:17.806571 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:15:17.806526 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls podName:105ecf64-8e04-4c31-8128-7af4855b2368 nodeName:}" failed. No retries permitted until 2026-07-06 17:15:18.306504988 +0000 UTC m=+2215.076396528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls") pod "maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" (UID: "105ecf64-8e04-4c31-8128-7af4855b2368") : secret "maas-api-e2e-inf-b-ebaa5d-serving-cert" not found Jul 06 17:15:17.807084 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.807060 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ecf64-8e04-4c31-8128-7af4855b2368-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.815885 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.815853 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxn2g\" (UniqueName: \"kubernetes.io/projected/105ecf64-8e04-4c31-8128-7af4855b2368-kube-api-access-sxn2g\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:17.916103 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:17.916067 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:18.311035 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:18.311000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:18.313688 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:18.313667 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls\") pod \"maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:18.319322 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:18.319281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:18.518179 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:18.518135 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:18.534397 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:18.534353 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:18.713363 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:18.713333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2"] Jul 06 17:15:18.715148 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:15:18.715121 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105ecf64_8e04_4c31_8128_7af4855b2368.slice/crio-19a3e01c4176a8082a9716ef85ef92b491ab9241b17db110451140b37a39fb1a WatchSource:0}: Error finding container 19a3e01c4176a8082a9716ef85ef92b491ab9241b17db110451140b37a39fb1a: Status 404 returned error can't find the container with id 19a3e01c4176a8082a9716ef85ef92b491ab9241b17db110451140b37a39fb1a Jul 06 17:15:19.470776 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:19.470670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" event={"ID":"105ecf64-8e04-4c31-8128-7af4855b2368","Type":"ContainerStarted","Data":"41e361ace197a8f11380399f54ec65e93a001f808149530c6ed87df8313a8b88"} Jul 06 17:15:19.471176 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:19.470835 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:19.471176 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:19.470869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" event={"ID":"105ecf64-8e04-4c31-8128-7af4855b2368","Type":"ContainerStarted","Data":"19a3e01c4176a8082a9716ef85ef92b491ab9241b17db110451140b37a39fb1a"} Jul 06 17:15:19.497337 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:19.497288 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" podStartSLOduration=2.141373055 podStartE2EDuration="2.497271629s" totalImagesPullingTime="355.898574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:15:19.495385394 +0000 UTC m=+2216.265276937" watchObservedRunningTime="2026-07-06 17:15:19.497271629 +0000 UTC m=+2216.267163170" Jul 06 17:15:22.049944 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:22.049907 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:24.452718 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:24.452670 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 17:15:24.453256 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:24.452769 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 17:15:25.482664 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:25.482638 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:15:25.503094 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:25.503064 2574 generic.go:356] "Generic (PLEG): container finished" podID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerID="786fed8f0ee235f51d75bbbcf31b7ac3612f365182f364e456f8967ca3323280" exitCode=0 Jul 06 17:15:25.503242 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:25.503135 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" event={"ID":"d175a1de-444f-4cf4-8516-d0b05d9e32bd","Type":"ContainerDied","Data":"786fed8f0ee235f51d75bbbcf31b7ac3612f365182f364e456f8967ca3323280"} Jul 06 17:15:25.503537 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:25.503523 2574 scope.go:117] "RemoveContainer" containerID="786fed8f0ee235f51d75bbbcf31b7ac3612f365182f364e456f8967ca3323280" Jul 06 17:15:25.504790 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:25.504763 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:25.505012 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:25.504904 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:25.505143 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:25.505047 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:26.509970 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:26.509938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" event={"ID":"d175a1de-444f-4cf4-8516-d0b05d9e32bd","Type":"ContainerStarted","Data":"58c0514db216bed1d7c629ab1472dd96c085a695b18cb365a876690063f0e8e4"} Jul 06 17:15:26.625051 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:26.625015 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:26.625398 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:26.625376 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 06 17:15:26.625449 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:26.625425 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 06 17:15:26.872963 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:26.872789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:27.624979 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:27.624941 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 06 17:15:27.625375 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:27.624994 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 06 17:15:28.624831 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:28.624736 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 06 17:15:28.624831 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:28.624797 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 06 17:15:29.625103 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:29.625064 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 06 17:15:29.625539 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:29.625129 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 06 17:15:30.628989 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:30.628956 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:30.629486 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:30.629305 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:30.630134 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:30.630118 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:15:30.919335 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:30.919259 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:31.019232 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:31.019180 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:31.540946 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:31.540916 2574 generic.go:356] "Generic (PLEG): container finished" podID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerID="61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9" exitCode=0 Jul 06 17:15:31.541157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:31.540988 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" event={"ID":"81158c40-bb7f-4785-841f-9a736ed31e2a","Type":"ContainerDied","Data":"61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9"} Jul 06 17:15:31.541549 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:31.541526 2574 scope.go:117] "RemoveContainer" containerID="61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9" Jul 06 17:15:31.542653 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:31.542621 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:31.542750 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:31.542698 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:31.542750 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:31.542733 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:15:32.459617 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:32.459570 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:32.552577 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:32.552511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" event={"ID":"81158c40-bb7f-4785-841f-9a736ed31e2a","Type":"ContainerStarted","Data":"fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2"} Jul 06 17:15:32.989775 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:32.989737 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:32.991097 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:32.991072 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 06 17:15:32.991157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:32.991136 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 06 17:15:33.989298 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:33.989259 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 06 17:15:33.989728 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:33.989327 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 06 17:15:34.990017 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:34.989978 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 06 17:15:34.990533 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:34.990058 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 06 17:15:35.989367 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:35.989324 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 06 17:15:35.989566 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:35.989381 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 06 17:15:36.994726 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:36.994697 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:36.995111 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:36.994968 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:36.995784 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:36.995763 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:15:39.257184 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:39.257145 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:39.358601 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:39.358560 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:55.980994 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:55.980960 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:15:59.431512 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:15:59.431480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:16:30.804368 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:30.804316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:16:31.781312 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:31.781275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:16:32.104660 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.104543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:16:32.496293 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.496257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:16:32.593757 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.593719 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:16:32.693056 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.693025 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7"] Jul 06 17:16:32.693313 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.693289 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" podUID="695307e6-64fe-4022-8fb2-379478f49065" containerName="maas-api" containerID="cri-o://1a4307e3496238127e93a3640813e8b757a93473df73136e97458db734093323" gracePeriod=30 Jul 06 17:16:32.735032 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.734990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:16:32.853619 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.853586 2574 generic.go:356] "Generic (PLEG): container finished" podID="695307e6-64fe-4022-8fb2-379478f49065" containerID="1a4307e3496238127e93a3640813e8b757a93473df73136e97458db734093323" exitCode=0 Jul 06 17:16:32.853783 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.853630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" event={"ID":"695307e6-64fe-4022-8fb2-379478f49065","Type":"ContainerDied","Data":"1a4307e3496238127e93a3640813e8b757a93473df73136e97458db734093323"} Jul 06 17:16:32.943176 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:32.943153 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:16:33.041903 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.041862 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695307e6-64fe-4022-8fb2-379478f49065-maas-api-tls\") pod \"695307e6-64fe-4022-8fb2-379478f49065\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " Jul 06 17:16:33.041903 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.041907 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rq2f\" (UniqueName: \"kubernetes.io/projected/695307e6-64fe-4022-8fb2-379478f49065-kube-api-access-5rq2f\") pod \"695307e6-64fe-4022-8fb2-379478f49065\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " Jul 06 17:16:33.042155 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.041943 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695307e6-64fe-4022-8fb2-379478f49065-openshift-service-ca\") pod \"695307e6-64fe-4022-8fb2-379478f49065\" (UID: \"695307e6-64fe-4022-8fb2-379478f49065\") " Jul 06 17:16:33.042347 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.042319 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695307e6-64fe-4022-8fb2-379478f49065-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "695307e6-64fe-4022-8fb2-379478f49065" (UID: "695307e6-64fe-4022-8fb2-379478f49065"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:16:33.044088 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.044060 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695307e6-64fe-4022-8fb2-379478f49065-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "695307e6-64fe-4022-8fb2-379478f49065" (UID: "695307e6-64fe-4022-8fb2-379478f49065"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:16:33.044196 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.044085 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695307e6-64fe-4022-8fb2-379478f49065-kube-api-access-5rq2f" (OuterVolumeSpecName: "kube-api-access-5rq2f") pod "695307e6-64fe-4022-8fb2-379478f49065" (UID: "695307e6-64fe-4022-8fb2-379478f49065"). InnerVolumeSpecName "kube-api-access-5rq2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:16:33.143899 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.143810 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695307e6-64fe-4022-8fb2-379478f49065-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:16:33.143899 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.143840 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rq2f\" (UniqueName: \"kubernetes.io/projected/695307e6-64fe-4022-8fb2-379478f49065-kube-api-access-5rq2f\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:16:33.143899 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.143850 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695307e6-64fe-4022-8fb2-379478f49065-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:16:33.859718 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.859683 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" Jul 06 17:16:33.859718 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.859712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7" event={"ID":"695307e6-64fe-4022-8fb2-379478f49065","Type":"ContainerDied","Data":"8a4eba961815fed53d3c6d882598b8bf78187f384e5c34176d48abaeb426f7ad"} Jul 06 17:16:33.859946 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.859770 2574 scope.go:117] "RemoveContainer" containerID="1a4307e3496238127e93a3640813e8b757a93473df73136e97458db734093323" Jul 06 17:16:33.877334 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.877306 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7"] Jul 06 17:16:33.881080 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:33.881045 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cdb744-6c66886f56-49zv7"] Jul 06 17:16:35.748088 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:35.748046 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695307e6-64fe-4022-8fb2-379478f49065" path="/var/lib/kubelet/pods/695307e6-64fe-4022-8fb2-379478f49065/volumes" Jul 06 17:16:37.539208 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:37.539175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:16:57.510464 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:16:57.510430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:14.622292 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:14.622257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:14.868912 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:14.868876 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs"] Jul 06 17:17:14.869114 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:14.869097 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" containerID="cri-o://58c0514db216bed1d7c629ab1472dd96c085a695b18cb365a876690063f0e8e4" gracePeriod=30 Jul 06 17:17:14.905759 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:14.905669 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:15.132684 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.132596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:15.295947 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.295913 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:15.632419 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.632332 2574 patch_prober.go:28] interesting pod/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 17:17:15.632882 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.632414 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 17:17:15.707892 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.707859 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:15.808302 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.808272 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:15.899531 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.899431 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2"] Jul 06 17:17:15.899691 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.899665 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" podUID="105ecf64-8e04-4c31-8128-7af4855b2368" containerName="maas-api" containerID="cri-o://41e361ace197a8f11380399f54ec65e93a001f808149530c6ed87df8313a8b88" gracePeriod=30 Jul 06 17:17:15.962945 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:15.962911 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:16.070642 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.070612 2574 generic.go:356] "Generic (PLEG): container finished" podID="105ecf64-8e04-4c31-8128-7af4855b2368" containerID="41e361ace197a8f11380399f54ec65e93a001f808149530c6ed87df8313a8b88" exitCode=0 Jul 06 17:17:16.070840 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.070695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" event={"ID":"105ecf64-8e04-4c31-8128-7af4855b2368","Type":"ContainerDied","Data":"41e361ace197a8f11380399f54ec65e93a001f808149530c6ed87df8313a8b88"} Jul 06 17:17:16.151938 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.151872 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:17:16.256256 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.256222 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ecf64-8e04-4c31-8128-7af4855b2368-openshift-service-ca\") pod \"105ecf64-8e04-4c31-8128-7af4855b2368\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " Jul 06 17:17:16.256493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.256331 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxn2g\" (UniqueName: \"kubernetes.io/projected/105ecf64-8e04-4c31-8128-7af4855b2368-kube-api-access-sxn2g\") pod \"105ecf64-8e04-4c31-8128-7af4855b2368\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " Jul 06 17:17:16.256493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.256373 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls\") pod \"105ecf64-8e04-4c31-8128-7af4855b2368\" (UID: \"105ecf64-8e04-4c31-8128-7af4855b2368\") " Jul 06 17:17:16.256724 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.256697 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105ecf64-8e04-4c31-8128-7af4855b2368-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "105ecf64-8e04-4c31-8128-7af4855b2368" (UID: "105ecf64-8e04-4c31-8128-7af4855b2368"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:17:16.258642 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.258604 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105ecf64-8e04-4c31-8128-7af4855b2368-kube-api-access-sxn2g" (OuterVolumeSpecName: "kube-api-access-sxn2g") pod "105ecf64-8e04-4c31-8128-7af4855b2368" (UID: "105ecf64-8e04-4c31-8128-7af4855b2368"). InnerVolumeSpecName "kube-api-access-sxn2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:17:16.258754 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.258646 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "105ecf64-8e04-4c31-8128-7af4855b2368" (UID: "105ecf64-8e04-4c31-8128-7af4855b2368"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:17:16.357178 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.357136 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxn2g\" (UniqueName: \"kubernetes.io/projected/105ecf64-8e04-4c31-8128-7af4855b2368-kube-api-access-sxn2g\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:16.357178 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.357174 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ecf64-8e04-4c31-8128-7af4855b2368-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:16.357178 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:16.357184 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ecf64-8e04-4c31-8128-7af4855b2368-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:17.076325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:17.076288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" event={"ID":"105ecf64-8e04-4c31-8128-7af4855b2368","Type":"ContainerDied","Data":"19a3e01c4176a8082a9716ef85ef92b491ab9241b17db110451140b37a39fb1a"} Jul 06 17:17:17.076325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:17.076305 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2" Jul 06 17:17:17.076852 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:17.076357 2574 scope.go:117] "RemoveContainer" containerID="41e361ace197a8f11380399f54ec65e93a001f808149530c6ed87df8313a8b88" Jul 06 17:17:17.102973 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:17.102941 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2"] Jul 06 17:17:17.106698 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:17.106665 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ebaa5d-bb557549-45nt2"] Jul 06 17:17:17.749034 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:17.749001 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105ecf64-8e04-4c31-8128-7af4855b2368" path="/var/lib/kubelet/pods/105ecf64-8e04-4c31-8128-7af4855b2368/volumes" Jul 06 17:17:20.094700 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.094663 2574 generic.go:356] "Generic (PLEG): container finished" podID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerID="58c0514db216bed1d7c629ab1472dd96c085a695b18cb365a876690063f0e8e4" exitCode=0 Jul 06 17:17:20.095189 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.094725 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" event={"ID":"d175a1de-444f-4cf4-8516-d0b05d9e32bd","Type":"ContainerDied","Data":"58c0514db216bed1d7c629ab1472dd96c085a695b18cb365a876690063f0e8e4"} Jul 06 17:17:20.095189 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.094788 2574 scope.go:117] "RemoveContainer" containerID="786fed8f0ee235f51d75bbbcf31b7ac3612f365182f364e456f8967ca3323280" Jul 06 17:17:20.128368 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.128343 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:17:20.188293 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188200 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-envoy\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188293 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188245 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-token\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188293 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188294 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-data\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188624 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188325 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istiod-ca-cert\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188624 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188347 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-credential-socket\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188624 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188390 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fksj\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-kube-api-access-8fksj\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188624 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188416 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-podinfo\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188624 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188493 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-socket\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188624 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188530 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-certs\") pod \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\" (UID: \"d175a1de-444f-4cf4-8516-d0b05d9e32bd\") " Jul 06 17:17:20.188923 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188773 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:17:20.188923 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188893 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:20.189023 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.188977 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:20.189075 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.189056 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:20.189138 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.189116 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:20.189325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.189300 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:20.190582 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.190563 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:17:20.190898 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.190863 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-kube-api-access-8fksj" (OuterVolumeSpecName: "kube-api-access-8fksj") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "kube-api-access-8fksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:17:20.190959 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.190937 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:20.190998 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.190953 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d175a1de-444f-4cf4-8516-d0b05d9e32bd" (UID: "d175a1de-444f-4cf4-8516-d0b05d9e32bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:17:20.289665 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.289622 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:20.289665 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.289656 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:20.289665 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.289666 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fksj\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-kube-api-access-8fksj\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:20.289665 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.289675 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:20.289938 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.289685 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:20.289938 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.289693 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:20.289938 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.289702 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:20.289938 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:20.289709 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d175a1de-444f-4cf4-8516-d0b05d9e32bd-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:21.100958 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:21.100924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" event={"ID":"d175a1de-444f-4cf4-8516-d0b05d9e32bd","Type":"ContainerDied","Data":"d527b1adcccd518f4a797a55d219fae7eb610f77199f5fb9ad855b4d59e68845"} Jul 06 17:17:21.101448 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:21.100973 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs" Jul 06 17:17:21.101448 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:21.100985 2574 scope.go:117] "RemoveContainer" containerID="58c0514db216bed1d7c629ab1472dd96c085a695b18cb365a876690063f0e8e4" Jul 06 17:17:21.128067 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:21.127940 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs"] Jul 06 17:17:21.130808 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:21.130783 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cdb744-openshift-default-f64ff589c-58cfs"] Jul 06 17:17:21.749316 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:21.749275 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" path="/var/lib/kubelet/pods/d175a1de-444f-4cf4-8516-d0b05d9e32bd/volumes" Jul 06 17:17:22.431365 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:22.431324 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:22.532036 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:22.532001 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:22.669698 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:22.669660 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs"] Jul 06 17:17:22.669930 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:22.669906 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" containerID="cri-o://fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2" gracePeriod=30 Jul 06 17:17:22.702121 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:22.702044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:22.936258 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:22.936224 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:23.290464 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.290433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:23.579509 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579412 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw"] Jul 06 17:17:23.579878 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579799 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695307e6-64fe-4022-8fb2-379478f49065" containerName="maas-api" Jul 06 17:17:23.579878 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579811 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="695307e6-64fe-4022-8fb2-379478f49065" containerName="maas-api" Jul 06 17:17:23.579878 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579820 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" Jul 06 17:17:23.579878 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579826 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" Jul 06 17:17:23.579878 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579834 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105ecf64-8e04-4c31-8128-7af4855b2368" containerName="maas-api" Jul 06 17:17:23.579878 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579840 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="105ecf64-8e04-4c31-8128-7af4855b2368" containerName="maas-api" Jul 06 17:17:23.579878 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579847 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" Jul 06 17:17:23.579878 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579852 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" Jul 06 17:17:23.580161 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579901 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="105ecf64-8e04-4c31-8128-7af4855b2368" containerName="maas-api" Jul 06 17:17:23.580161 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579911 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="695307e6-64fe-4022-8fb2-379478f49065" containerName="maas-api" Jul 06 17:17:23.580161 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579918 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" Jul 06 17:17:23.580161 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.579923 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d175a1de-444f-4cf4-8516-d0b05d9e32bd" containerName="istio-proxy" Jul 06 17:17:23.582842 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.582826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.587131 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.587112 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ce033f-openshift-default-dockercfg-ztjh8\"" Jul 06 17:17:23.603750 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.603704 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw"] Jul 06 17:17:23.618525 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-data\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.618688 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618530 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:23.618688 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-podinfo\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.618688 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-socket\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.618688 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9wc\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-kube-api-access-fh9wc\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.618888 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-token\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.618888 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-certs\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.618888 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istiod-ca-cert\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.619039 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-envoy\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.619039 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.618990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-credential-socket\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.719633 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.719593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istiod-ca-cert\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.719788 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.719692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-envoy\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.719788 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.719718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-credential-socket\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.719788 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.719750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-data\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.719788 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.719782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-podinfo\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720024 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.719809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-socket\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720024 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.719839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9wc\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-kube-api-access-fh9wc\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720024 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.719878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-token\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720183 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.720053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-certs\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720183 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.720129 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-credential-socket\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720183 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.720172 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-data\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720317 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.720215 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-socket\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720406 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.720387 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istiod-ca-cert\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.720509 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.720415 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-certs\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.722330 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.722306 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-podinfo\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.722448 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.722328 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-envoy\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.727743 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.727716 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9wc\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-kube-api-access-fh9wc\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.727743 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.727730 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-token\") pod \"e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:23.814208 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.814172 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:23.896624 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.896540 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ce033f-openshift-default-dockercfg-ztjh8\"" Jul 06 17:17:23.904737 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:23.904716 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:24.050481 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:24.050440 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw"] Jul 06 17:17:24.052556 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:17:24.052528 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db4afad_b278_4ec5_9ff8_2095f8d7fed1.slice/crio-e9e10b13b46280f3e2b530c8fc06a4c9ec4a67f631e896c423eb39f34cc1b9d6 WatchSource:0}: Error finding container e9e10b13b46280f3e2b530c8fc06a4c9ec4a67f631e896c423eb39f34cc1b9d6: Status 404 returned error can't find the container with id e9e10b13b46280f3e2b530c8fc06a4c9ec4a67f631e896c423eb39f34cc1b9d6 Jul 06 17:17:24.054647 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:24.054616 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:24.054716 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:24.054682 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:24.054716 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:24.054710 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:24.117717 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:24.117687 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" event={"ID":"6db4afad-b278-4ec5-9ff8-2095f8d7fed1","Type":"ContainerStarted","Data":"a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5"} Jul 06 17:17:24.117887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:24.117742 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" event={"ID":"6db4afad-b278-4ec5-9ff8-2095f8d7fed1","Type":"ContainerStarted","Data":"e9e10b13b46280f3e2b530c8fc06a4c9ec4a67f631e896c423eb39f34cc1b9d6"} Jul 06 17:17:24.138938 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:24.138882 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" podStartSLOduration=1.138860592 podStartE2EDuration="1.138860592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:17:24.138118146 +0000 UTC m=+2340.908009686" watchObservedRunningTime="2026-07-06 17:17:24.138860592 +0000 UTC m=+2340.908752131" Jul 06 17:17:24.904870 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:24.904839 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:25.910687 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:25.910651 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:26.128543 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.128505 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:26.129780 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.129756 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:26.193307 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.193227 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:26.265471 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.265430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:26.297610 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.297576 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m"] Jul 06 17:17:26.300484 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.300447 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.302907 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.302885 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-ce033f-serving-cert\"" Jul 06 17:17:26.311414 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.311388 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m"] Jul 06 17:17:26.342999 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.342970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59857777-bc08-47f7-98e4-8c4c50332159-openshift-service-ca\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.343157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.343029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvhh\" (UniqueName: \"kubernetes.io/projected/59857777-bc08-47f7-98e4-8c4c50332159-kube-api-access-4qvhh\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.343157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.343065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.350680 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.350650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:26.443789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.443682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59857777-bc08-47f7-98e4-8c4c50332159-openshift-service-ca\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.443789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.443761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvhh\" (UniqueName: \"kubernetes.io/projected/59857777-bc08-47f7-98e4-8c4c50332159-kube-api-access-4qvhh\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.444036 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.443808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.444036 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:17:26.443945 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-ce033f-serving-cert: secret "maas-api-e2e-api-a-ce033f-serving-cert" not found Jul 06 17:17:26.444036 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:17:26.444012 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls podName:59857777-bc08-47f7-98e4-8c4c50332159 nodeName:}" failed. No retries permitted until 2026-07-06 17:17:26.943990238 +0000 UTC m=+2343.713881769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls") pod "maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" (UID: "59857777-bc08-47f7-98e4-8c4c50332159") : secret "maas-api-e2e-api-a-ce033f-serving-cert" not found Jul 06 17:17:26.444315 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.444294 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59857777-bc08-47f7-98e4-8c4c50332159-openshift-service-ca\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.452120 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.452095 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvhh\" (UniqueName: \"kubernetes.io/projected/59857777-bc08-47f7-98e4-8c4c50332159-kube-api-access-4qvhh\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.897291 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.897256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:26.947426 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.947393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:26.947812 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:17:26.947573 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-ce033f-serving-cert: secret "maas-api-e2e-api-a-ce033f-serving-cert" not found Jul 06 17:17:26.947812 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:17:26.947636 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls podName:59857777-bc08-47f7-98e4-8c4c50332159 nodeName:}" failed. No retries permitted until 2026-07-06 17:17:27.947619457 +0000 UTC m=+2344.717510982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls") pod "maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" (UID: "59857777-bc08-47f7-98e4-8c4c50332159") : secret "maas-api-e2e-api-a-ce033f-serving-cert" not found Jul 06 17:17:26.992755 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:26.992724 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:27.110475 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.110432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:27.919990 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.919964 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:17:27.957277 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957185 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-envoy\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957277 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957235 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-data\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957277 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957261 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-socket\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957300 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-credential-socket\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957359 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-token\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957408 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81158c40-bb7f-4785-841f-9a736ed31e2a-istiod-ca-cert\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957465 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2ccq\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-kube-api-access-r2ccq\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957510 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-podinfo\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957575 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-certs\") pod \"81158c40-bb7f-4785-841f-9a736ed31e2a\" (UID: \"81158c40-bb7f-4785-841f-9a736ed31e2a\") " Jul 06 17:17:27.957887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957682 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:27.957887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957789 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:27.958305 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957890 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:27.958305 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.957951 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:27.958305 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.958105 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81158c40-bb7f-4785-841f-9a736ed31e2a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:17:27.958305 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.958154 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:27.958305 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.958174 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:27.958305 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.958188 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:27.958305 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.958205 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-data" (OuterVolumeSpecName: "istio-data") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:27.959958 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.959931 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:17:27.960076 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.960063 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-kube-api-access-r2ccq" (OuterVolumeSpecName: "kube-api-access-r2ccq") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "kube-api-access-r2ccq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:17:27.960250 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.960218 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-token" (OuterVolumeSpecName: "istio-token") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:17:27.960409 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.960391 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81158c40-bb7f-4785-841f-9a736ed31e2a" (UID: "81158c40-bb7f-4785-841f-9a736ed31e2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:17:27.961007 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:27.960985 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls\") pod \"maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:28.059685 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.059652 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:28.059685 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.059680 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81158c40-bb7f-4785-841f-9a736ed31e2a-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:28.059685 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.059690 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2ccq\" (UniqueName: \"kubernetes.io/projected/81158c40-bb7f-4785-841f-9a736ed31e2a-kube-api-access-r2ccq\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:28.059925 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.059699 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:28.059925 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.059707 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:28.059925 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.059716 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81158c40-bb7f-4785-841f-9a736ed31e2a-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:28.112436 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.112379 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:28.140611 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.140561 2574 generic.go:356] "Generic (PLEG): container finished" podID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerID="fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2" exitCode=0 Jul 06 17:17:28.140771 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.140642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" event={"ID":"81158c40-bb7f-4785-841f-9a736ed31e2a","Type":"ContainerDied","Data":"fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2"} Jul 06 17:17:28.140771 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.140686 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" Jul 06 17:17:28.140771 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.140726 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs" event={"ID":"81158c40-bb7f-4785-841f-9a736ed31e2a","Type":"ContainerDied","Data":"cf3686ec428736be1291346801536ecac1e360050047622e1663c65a5eae8570"} Jul 06 17:17:28.140771 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.140749 2574 scope.go:117] "RemoveContainer" containerID="fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2" Jul 06 17:17:28.151222 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.151203 2574 scope.go:117] "RemoveContainer" containerID="61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9" Jul 06 17:17:28.159936 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.159889 2574 scope.go:117] "RemoveContainer" containerID="fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2" Jul 06 17:17:28.160235 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:17:28.160211 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2\": container with ID starting with fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2 not found: ID does not exist" containerID="fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2" Jul 06 17:17:28.160286 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.160248 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2"} err="failed to get container status \"fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2\": rpc error: code = NotFound desc = could not find container \"fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2\": container with ID starting with fb7a9e1141e8fe39a5a7809ffd155df748a3a32aad02ec1b56915b4eb30ff3c2 not found: ID does not exist" Jul 06 17:17:28.160286 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.160273 2574 scope.go:117] "RemoveContainer" containerID="61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9" Jul 06 17:17:28.160628 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:17:28.160602 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9\": container with ID starting with 61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9 not found: ID does not exist" containerID="61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9" Jul 06 17:17:28.160699 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.160642 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9"} err="failed to get container status \"61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9\": rpc error: code = NotFound desc = could not find container \"61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9\": container with ID starting with 61f6b3b5dc730ba041b66244f08e79026c64cf7809bc89874383a8bbf5be91f9 not found: ID does not exist" Jul 06 17:17:28.168360 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.168334 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs"] Jul 06 17:17:28.170448 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.170419 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ebaa5d-openshift-default-67b687fb47-smjfs"] Jul 06 17:17:28.252172 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:28.252148 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m"] Jul 06 17:17:28.254897 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:17:28.254861 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59857777_bc08_47f7_98e4_8c4c50332159.slice/crio-8fd396cf4a29b761b0ffb774db1667a8c359b50c1512b711d5ffeb3903f72b3e WatchSource:0}: Error finding container 8fd396cf4a29b761b0ffb774db1667a8c359b50c1512b711d5ffeb3903f72b3e: Status 404 returned error can't find the container with id 8fd396cf4a29b761b0ffb774db1667a8c359b50c1512b711d5ffeb3903f72b3e Jul 06 17:17:29.147731 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:29.147700 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" event={"ID":"59857777-bc08-47f7-98e4-8c4c50332159","Type":"ContainerStarted","Data":"5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74"} Jul 06 17:17:29.148168 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:29.147749 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" event={"ID":"59857777-bc08-47f7-98e4-8c4c50332159","Type":"ContainerStarted","Data":"8fd396cf4a29b761b0ffb774db1667a8c359b50c1512b711d5ffeb3903f72b3e"} Jul 06 17:17:29.148168 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:29.147815 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:29.165993 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:29.165945 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" podStartSLOduration=2.69810981 podStartE2EDuration="3.165930287s" totalImagesPullingTime="467.820477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:17:29.164710737 +0000 UTC m=+2345.934602278" watchObservedRunningTime="2026-07-06 17:17:29.165930287 +0000 UTC m=+2345.935821829" Jul 06 17:17:29.747951 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:29.747917 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" path="/var/lib/kubelet/pods/81158c40-bb7f-4785-841f-9a736ed31e2a/volumes" Jul 06 17:17:30.692087 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:30.692054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:31.214058 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.214025 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:31.597412 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.597372 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc"] Jul 06 17:17:31.597789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.597775 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" Jul 06 17:17:31.597846 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.597791 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" Jul 06 17:17:31.597846 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.597807 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" Jul 06 17:17:31.597846 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.597812 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" Jul 06 17:17:31.597971 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.597882 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" Jul 06 17:17:31.601073 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.601057 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.604246 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.604221 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9c929f-openshift-default-dockercfg-fx89r\"" Jul 06 17:17:31.624383 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.624346 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc"] Jul 06 17:17:31.633285 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.633257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:31.692675 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.692637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-certs\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.693128 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.692693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-socket\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.693128 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.692773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-credential-socket\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.693128 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.692817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-envoy\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.693128 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.692876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-istio-token\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.693128 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.692923 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c00122-e6d5-43e5-8869-12def26a6934-istio-podinfo\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.693128 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.692945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-data\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.693128 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.693010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c00122-e6d5-43e5-8869-12def26a6934-istiod-ca-cert\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.693128 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.693083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8bs\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-kube-api-access-hc8bs\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794059 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8bs\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-kube-api-access-hc8bs\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794059 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-certs\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-socket\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-credential-socket\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794325 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-envoy\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794498 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-istio-token\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794498 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c00122-e6d5-43e5-8869-12def26a6934-istio-podinfo\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794616 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-data\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794616 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794515 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-certs\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794616 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c00122-e6d5-43e5-8869-12def26a6934-istiod-ca-cert\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794760 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794729 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-credential-socket\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794834 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794757 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-socket\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.794886 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.794856 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-data\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.795255 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.795232 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c00122-e6d5-43e5-8869-12def26a6934-istiod-ca-cert\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.796642 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.796625 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-envoy\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.796856 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.796837 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c00122-e6d5-43e5-8869-12def26a6934-istio-podinfo\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.806025 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.806001 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8bs\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-kube-api-access-hc8bs\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.806143 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.806126 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-istio-token\") pod \"e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:31.911751 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:31.911654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:32.014766 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.014735 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:32.060790 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.060766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc"] Jul 06 17:17:32.062311 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:17:32.062287 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c00122_e6d5_43e5_8869_12def26a6934.slice/crio-b421fc7af8f22521bdb83419b5753c17c6f3900c6c4fe71810d0156f0b173ae5 WatchSource:0}: Error finding container b421fc7af8f22521bdb83419b5753c17c6f3900c6c4fe71810d0156f0b173ae5: Status 404 returned error can't find the container with id b421fc7af8f22521bdb83419b5753c17c6f3900c6c4fe71810d0156f0b173ae5 Jul 06 17:17:32.064364 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.064332 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:32.064484 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.064408 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:32.064484 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.064449 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:32.164557 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.164438 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" event={"ID":"b1c00122-e6d5-43e5-8869-12def26a6934","Type":"ContainerStarted","Data":"7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be"} Jul 06 17:17:32.164557 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.164530 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" event={"ID":"b1c00122-e6d5-43e5-8869-12def26a6934","Type":"ContainerStarted","Data":"b421fc7af8f22521bdb83419b5753c17c6f3900c6c4fe71810d0156f0b173ae5"} Jul 06 17:17:32.192227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.192168 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" podStartSLOduration=1.192146192 podStartE2EDuration="1.192146192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:17:32.190809492 +0000 UTC m=+2348.960701033" watchObservedRunningTime="2026-07-06 17:17:32.192146192 +0000 UTC m=+2348.962037734" Jul 06 17:17:32.912170 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:32.912119 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:33.916858 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:33.916821 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:34.181401 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.181306 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:34.182422 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.182404 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:34.283943 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.283900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:34.352671 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.352639 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5"] Jul 06 17:17:34.353094 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.353082 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="81158c40-bb7f-4785-841f-9a736ed31e2a" containerName="istio-proxy" Jul 06 17:17:34.356297 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.356276 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.360134 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.360107 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-9c929f-serving-cert\"" Jul 06 17:17:34.363104 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.363077 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5"] Jul 06 17:17:34.390242 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.390206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:34.421973 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.421936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2466a4a-c778-4152-b02b-cf3a4647baee-maas-api-tls\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.421973 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.421973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq2f6\" (UniqueName: \"kubernetes.io/projected/a2466a4a-c778-4152-b02b-cf3a4647baee-kube-api-access-jq2f6\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.422172 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.422040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2466a4a-c778-4152-b02b-cf3a4647baee-openshift-service-ca\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.500002 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.499970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:34.523297 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.523259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2466a4a-c778-4152-b02b-cf3a4647baee-maas-api-tls\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.523297 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.523298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq2f6\" (UniqueName: \"kubernetes.io/projected/a2466a4a-c778-4152-b02b-cf3a4647baee-kube-api-access-jq2f6\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.523539 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.523486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2466a4a-c778-4152-b02b-cf3a4647baee-openshift-service-ca\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.524060 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.524039 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2466a4a-c778-4152-b02b-cf3a4647baee-openshift-service-ca\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.525695 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.525674 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2466a4a-c778-4152-b02b-cf3a4647baee-maas-api-tls\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.548427 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.548381 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq2f6\" (UniqueName: \"kubernetes.io/projected/a2466a4a-c778-4152-b02b-cf3a4647baee-kube-api-access-jq2f6\") pod \"maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.595516 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.595482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:34.668333 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.668296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:34.855247 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:34.855212 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5"] Jul 06 17:17:34.859079 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:17:34.859050 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2466a4a_c778_4152_b02b_cf3a4647baee.slice/crio-b6ac63ca7eb71d36cc2c371390bb91d6b57f0dcc852f16c122e6dddf67cc322d WatchSource:0}: Error finding container b6ac63ca7eb71d36cc2c371390bb91d6b57f0dcc852f16c122e6dddf67cc322d: Status 404 returned error can't find the container with id b6ac63ca7eb71d36cc2c371390bb91d6b57f0dcc852f16c122e6dddf67cc322d Jul 06 17:17:35.159091 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:35.159014 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:35.186411 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:35.186372 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" event={"ID":"a2466a4a-c778-4152-b02b-cf3a4647baee","Type":"ContainerStarted","Data":"b6ac63ca7eb71d36cc2c371390bb91d6b57f0dcc852f16c122e6dddf67cc322d"} Jul 06 17:17:35.342074 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:35.341948 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:35.458908 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:35.458820 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:36.192432 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:36.192394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" event={"ID":"a2466a4a-c778-4152-b02b-cf3a4647baee","Type":"ContainerStarted","Data":"c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5"} Jul 06 17:17:36.212118 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:36.212067 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" podStartSLOduration=1.799823648 podStartE2EDuration="2.21205241s" totalImagesPullingTime="412.228762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:17:36.211188093 +0000 UTC m=+2352.981079635" watchObservedRunningTime="2026-07-06 17:17:36.21205241 +0000 UTC m=+2352.981944003" Jul 06 17:17:37.197362 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:37.197333 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:38.816612 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:38.816570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:41.130548 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.130151 2574 patch_prober.go:28] interesting pod/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 17:17:41.130548 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.130219 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 17:17:41.219560 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.219471 2574 generic.go:356] "Generic (PLEG): container finished" podID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerID="a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5" exitCode=0 Jul 06 17:17:41.219560 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.219485 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" event={"ID":"6db4afad-b278-4ec5-9ff8-2095f8d7fed1","Type":"ContainerDied","Data":"a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5"} Jul 06 17:17:41.219920 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.219899 2574 scope.go:117] "RemoveContainer" containerID="a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5" Jul 06 17:17:41.220948 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.220924 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:41.221014 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.220988 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:41.221061 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.221019 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:41.234064 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.234012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:41.353071 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:41.352906 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:42.225973 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:42.225942 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" event={"ID":"6db4afad-b278-4ec5-9ff8-2095f8d7fed1","Type":"ContainerStarted","Data":"a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1"} Jul 06 17:17:42.905205 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:42.905171 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:42.905390 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:42.905357 2574 patch_prober.go:28] interesting pod/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 17:17:42.905530 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:42.905410 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 17:17:43.207881 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:43.207810 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:43.905904 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:43.905863 2574 patch_prober.go:28] interesting pod/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 17:17:43.906335 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:43.905917 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 17:17:44.905928 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:44.905872 2574 patch_prober.go:28] interesting pod/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 17:17:44.906372 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:44.905939 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 17:17:45.905981 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:45.905943 2574 patch_prober.go:28] interesting pod/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 17:17:45.906614 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:45.906001 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 17:17:46.909527 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:46.909496 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:46.909924 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:46.909839 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:46.910633 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:46.910612 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:17:47.086117 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:47.086086 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:47.522160 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:47.522128 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:47.929270 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:47.929221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:48.267479 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.267431 2574 generic.go:356] "Generic (PLEG): container finished" podID="b1c00122-e6d5-43e5-8869-12def26a6934" containerID="7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be" exitCode=0 Jul 06 17:17:48.267656 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.267513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" event={"ID":"b1c00122-e6d5-43e5-8869-12def26a6934","Type":"ContainerDied","Data":"7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be"} Jul 06 17:17:48.267987 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.267973 2574 scope.go:117] "RemoveContainer" containerID="7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be" Jul 06 17:17:48.268895 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.268871 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:48.268949 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.268933 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:48.268988 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.268965 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:17:48.322695 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.322668 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m"] Jul 06 17:17:48.322881 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.322860 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" podUID="59857777-bc08-47f7-98e4-8c4c50332159" containerName="maas-api" containerID="cri-o://5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74" gracePeriod=30 Jul 06 17:17:48.576078 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.576051 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:48.685374 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.685335 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qvhh\" (UniqueName: \"kubernetes.io/projected/59857777-bc08-47f7-98e4-8c4c50332159-kube-api-access-4qvhh\") pod \"59857777-bc08-47f7-98e4-8c4c50332159\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " Jul 06 17:17:48.685566 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.685425 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls\") pod \"59857777-bc08-47f7-98e4-8c4c50332159\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " Jul 06 17:17:48.685566 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.685479 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59857777-bc08-47f7-98e4-8c4c50332159-openshift-service-ca\") pod \"59857777-bc08-47f7-98e4-8c4c50332159\" (UID: \"59857777-bc08-47f7-98e4-8c4c50332159\") " Jul 06 17:17:48.685928 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.685889 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59857777-bc08-47f7-98e4-8c4c50332159-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59857777-bc08-47f7-98e4-8c4c50332159" (UID: "59857777-bc08-47f7-98e4-8c4c50332159"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:17:48.687572 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.687542 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59857777-bc08-47f7-98e4-8c4c50332159" (UID: "59857777-bc08-47f7-98e4-8c4c50332159"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:17:48.687660 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.687571 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59857777-bc08-47f7-98e4-8c4c50332159-kube-api-access-4qvhh" (OuterVolumeSpecName: "kube-api-access-4qvhh") pod "59857777-bc08-47f7-98e4-8c4c50332159" (UID: "59857777-bc08-47f7-98e4-8c4c50332159"). InnerVolumeSpecName "kube-api-access-4qvhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:17:48.786735 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.786700 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59857777-bc08-47f7-98e4-8c4c50332159-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:48.786735 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.786727 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59857777-bc08-47f7-98e4-8c4c50332159-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:48.786735 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:48.786737 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qvhh\" (UniqueName: \"kubernetes.io/projected/59857777-bc08-47f7-98e4-8c4c50332159-kube-api-access-4qvhh\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:49.181400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.181351 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:49.273579 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.273485 2574 generic.go:356] "Generic (PLEG): container finished" podID="59857777-bc08-47f7-98e4-8c4c50332159" containerID="5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74" exitCode=0 Jul 06 17:17:49.273579 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.273530 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" Jul 06 17:17:49.273813 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.273580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" event={"ID":"59857777-bc08-47f7-98e4-8c4c50332159","Type":"ContainerDied","Data":"5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74"} Jul 06 17:17:49.273813 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.273643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m" event={"ID":"59857777-bc08-47f7-98e4-8c4c50332159","Type":"ContainerDied","Data":"8fd396cf4a29b761b0ffb774db1667a8c359b50c1512b711d5ffeb3903f72b3e"} Jul 06 17:17:49.273813 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.273673 2574 scope.go:117] "RemoveContainer" containerID="5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74" Jul 06 17:17:49.275836 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.275812 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" event={"ID":"b1c00122-e6d5-43e5-8869-12def26a6934","Type":"ContainerStarted","Data":"f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a"} Jul 06 17:17:49.282716 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.282700 2574 scope.go:117] "RemoveContainer" containerID="5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74" Jul 06 17:17:49.282971 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:17:49.282951 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74\": container with ID starting with 5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74 not found: ID does not exist" containerID="5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74" Jul 06 17:17:49.283039 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.282978 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74"} err="failed to get container status \"5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74\": rpc error: code = NotFound desc = could not find container \"5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74\": container with ID starting with 5c3ea35806bf59fe85bbd5b25b2758dd85cd872e596e0a798ab38fa76a924c74 not found: ID does not exist" Jul 06 17:17:49.372403 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.372364 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m"] Jul 06 17:17:49.381917 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.381890 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ce033f-56fb95b4c5-l4t5m"] Jul 06 17:17:49.756575 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.756544 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59857777-bc08-47f7-98e4-8c4c50332159" path="/var/lib/kubelet/pods/59857777-bc08-47f7-98e4-8c4c50332159/volumes" Jul 06 17:17:49.787587 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.787520 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:49.912017 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.911983 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:49.913164 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.913137 2574 patch_prober.go:28] interesting pod/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 06 17:17:49.913304 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:49.913189 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 06 17:17:50.912700 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:50.912668 2574 patch_prober.go:28] interesting pod/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 06 17:17:50.913117 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:50.912724 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 06 17:17:51.912934 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:51.912841 2574 patch_prober.go:28] interesting pod/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 06 17:17:51.912934 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:51.912899 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 06 17:17:52.912944 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:52.912908 2574 patch_prober.go:28] interesting pod/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 06 17:17:52.913316 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:52.912958 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 06 17:17:53.912929 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:53.912896 2574 patch_prober.go:28] interesting pod/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 06 17:17:53.913107 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:53.912947 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 06 17:17:54.507686 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:54.507645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:54.606039 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:54.606004 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:54.771421 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:54.771331 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw"] Jul 06 17:17:54.771608 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:54.771585 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" containerID="cri-o://a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1" gracePeriod=30 Jul 06 17:17:54.824007 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:54.823972 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:54.915620 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:54.915592 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:54.915996 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:54.915874 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:54.916613 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:54.916596 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:17:55.025415 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:55.025221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:55.205717 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:55.205683 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:55.606570 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:55.606536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:55.752565 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:55.752526 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:55.832474 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:55.832422 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:56.010260 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.010228 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5"] Jul 06 17:17:56.010728 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.010500 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" podUID="a2466a4a-c778-4152-b02b-cf3a4647baee" containerName="maas-api" containerID="cri-o://c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5" gracePeriod=30 Jul 06 17:17:56.052826 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.052796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:17:56.257573 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.257549 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:56.313923 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.313834 2574 generic.go:356] "Generic (PLEG): container finished" podID="a2466a4a-c778-4152-b02b-cf3a4647baee" containerID="c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5" exitCode=0 Jul 06 17:17:56.313923 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.313878 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" Jul 06 17:17:56.314117 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.313929 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" event={"ID":"a2466a4a-c778-4152-b02b-cf3a4647baee","Type":"ContainerDied","Data":"c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5"} Jul 06 17:17:56.314117 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.313990 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5" event={"ID":"a2466a4a-c778-4152-b02b-cf3a4647baee","Type":"ContainerDied","Data":"b6ac63ca7eb71d36cc2c371390bb91d6b57f0dcc852f16c122e6dddf67cc322d"} Jul 06 17:17:56.314117 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.314026 2574 scope.go:117] "RemoveContainer" containerID="c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5" Jul 06 17:17:56.322401 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.322380 2574 scope.go:117] "RemoveContainer" containerID="c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5" Jul 06 17:17:56.322712 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:17:56.322692 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5\": container with ID starting with c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5 not found: ID does not exist" containerID="c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5" Jul 06 17:17:56.322817 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.322717 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5"} err="failed to get container status \"c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5\": rpc error: code = NotFound desc = could not find container \"c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5\": container with ID starting with c31fd862034abafc121e224df7a219f24a887a2b732b8ade502d33e5ff8bace5 not found: ID does not exist" Jul 06 17:17:56.357077 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.357052 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq2f6\" (UniqueName: \"kubernetes.io/projected/a2466a4a-c778-4152-b02b-cf3a4647baee-kube-api-access-jq2f6\") pod \"a2466a4a-c778-4152-b02b-cf3a4647baee\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " Jul 06 17:17:56.357227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.357094 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2466a4a-c778-4152-b02b-cf3a4647baee-maas-api-tls\") pod \"a2466a4a-c778-4152-b02b-cf3a4647baee\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " Jul 06 17:17:56.357227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.357110 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2466a4a-c778-4152-b02b-cf3a4647baee-openshift-service-ca\") pod \"a2466a4a-c778-4152-b02b-cf3a4647baee\" (UID: \"a2466a4a-c778-4152-b02b-cf3a4647baee\") " Jul 06 17:17:56.357565 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.357539 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2466a4a-c778-4152-b02b-cf3a4647baee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2466a4a-c778-4152-b02b-cf3a4647baee" (UID: "a2466a4a-c778-4152-b02b-cf3a4647baee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:17:56.359164 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.359141 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2466a4a-c778-4152-b02b-cf3a4647baee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2466a4a-c778-4152-b02b-cf3a4647baee" (UID: "a2466a4a-c778-4152-b02b-cf3a4647baee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:17:56.359271 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.359228 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2466a4a-c778-4152-b02b-cf3a4647baee-kube-api-access-jq2f6" (OuterVolumeSpecName: "kube-api-access-jq2f6") pod "a2466a4a-c778-4152-b02b-cf3a4647baee" (UID: "a2466a4a-c778-4152-b02b-cf3a4647baee"). InnerVolumeSpecName "kube-api-access-jq2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:17:56.457960 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.457926 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq2f6\" (UniqueName: \"kubernetes.io/projected/a2466a4a-c778-4152-b02b-cf3a4647baee-kube-api-access-jq2f6\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:56.457960 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.457956 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2466a4a-c778-4152-b02b-cf3a4647baee-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:56.457960 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.457966 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2466a4a-c778-4152-b02b-cf3a4647baee-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:17:56.645283 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.645243 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5"] Jul 06 17:17:56.653305 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:56.653278 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-9c929f-8d46b6db6-8t9t5"] Jul 06 17:17:57.748723 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:17:57.748688 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2466a4a-c778-4152-b02b-cf3a4647baee" path="/var/lib/kubelet/pods/a2466a4a-c778-4152-b02b-cf3a4647baee/volumes" Jul 06 17:18:00.021782 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.021758 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:18:00.194077 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.193987 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-token\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194077 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194031 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-envoy\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194077 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194066 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-socket\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194098 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-data\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194125 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-podinfo\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194140 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-certs\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194165 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh9wc\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-kube-api-access-fh9wc\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194206 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-credential-socket\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194400 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194244 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istiod-ca-cert\") pod \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\" (UID: \"6db4afad-b278-4ec5-9ff8-2095f8d7fed1\") " Jul 06 17:18:00.194765 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194545 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:00.194765 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194692 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.194874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194834 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:00.194932 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194899 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:18:00.195000 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.194927 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:00.195209 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.195180 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-data" (OuterVolumeSpecName: "istio-data") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:00.196394 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.196367 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-token" (OuterVolumeSpecName: "istio-token") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:18:00.196394 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.196364 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:18:00.196394 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.196369 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:00.196633 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.196612 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-kube-api-access-fh9wc" (OuterVolumeSpecName: "kube-api-access-fh9wc") pod "6db4afad-b278-4ec5-9ff8-2095f8d7fed1" (UID: "6db4afad-b278-4ec5-9ff8-2095f8d7fed1"). InnerVolumeSpecName "kube-api-access-fh9wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:18:00.295238 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.295200 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.295238 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.295233 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.295238 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.295244 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.295523 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.295254 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh9wc\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-kube-api-access-fh9wc\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.295523 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.295264 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.295523 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.295272 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.295523 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.295280 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.295523 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.295287 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4afad-b278-4ec5-9ff8-2095f8d7fed1-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:00.335346 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.335313 2574 generic.go:356] "Generic (PLEG): container finished" podID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerID="a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1" exitCode=0 Jul 06 17:18:00.335544 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.335358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" event={"ID":"6db4afad-b278-4ec5-9ff8-2095f8d7fed1","Type":"ContainerDied","Data":"a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1"} Jul 06 17:18:00.335544 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.335380 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" Jul 06 17:18:00.335544 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.335426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw" event={"ID":"6db4afad-b278-4ec5-9ff8-2095f8d7fed1","Type":"ContainerDied","Data":"e9e10b13b46280f3e2b530c8fc06a4c9ec4a67f631e896c423eb39f34cc1b9d6"} Jul 06 17:18:00.335544 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.335448 2574 scope.go:117] "RemoveContainer" containerID="a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1" Jul 06 17:18:00.344428 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.344408 2574 scope.go:117] "RemoveContainer" containerID="a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5" Jul 06 17:18:00.352726 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.352707 2574 scope.go:117] "RemoveContainer" containerID="a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1" Jul 06 17:18:00.352991 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:00.352971 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1\": container with ID starting with a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1 not found: ID does not exist" containerID="a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1" Jul 06 17:18:00.353036 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.353000 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1"} err="failed to get container status \"a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1\": rpc error: code = NotFound desc = could not find container \"a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1\": container with ID starting with a78e9bd131a0110b87c7955b0bdc0ef4f3012975a7b54a9c206d5ff8550c99e1 not found: ID does not exist" Jul 06 17:18:00.353036 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.353019 2574 scope.go:117] "RemoveContainer" containerID="a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5" Jul 06 17:18:00.353269 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:00.353251 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5\": container with ID starting with a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5 not found: ID does not exist" containerID="a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5" Jul 06 17:18:00.353312 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.353275 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5"} err="failed to get container status \"a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5\": rpc error: code = NotFound desc = could not find container \"a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5\": container with ID starting with a639f0a7fa968234c432402c8d9da8532597e1fec8c7fbb04ab4f6aca58e07e5 not found: ID does not exist" Jul 06 17:18:00.357403 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.357368 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw"] Jul 06 17:18:00.363569 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:00.363547 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ce033f-openshift-default-5f6f468755-vdjrw"] Jul 06 17:18:01.749442 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:01.749408 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" path="/var/lib/kubelet/pods/6db4afad-b278-4ec5-9ff8-2095f8d7fed1/volumes" Jul 06 17:18:02.800885 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:02.800841 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:02.900475 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:02.900433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:03.087534 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.087432 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc"] Jul 06 17:18:03.087728 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.087702 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" containerID="cri-o://f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a" gracePeriod=30 Jul 06 17:18:03.107299 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.107271 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:03.322290 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.322255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:03.653891 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.653849 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:03.925587 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925506 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx"] Jul 06 17:18:03.925952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925862 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59857777-bc08-47f7-98e4-8c4c50332159" containerName="maas-api" Jul 06 17:18:03.925952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925874 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="59857777-bc08-47f7-98e4-8c4c50332159" containerName="maas-api" Jul 06 17:18:03.925952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925889 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2466a4a-c778-4152-b02b-cf3a4647baee" containerName="maas-api" Jul 06 17:18:03.925952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925894 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2466a4a-c778-4152-b02b-cf3a4647baee" containerName="maas-api" Jul 06 17:18:03.925952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925908 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" Jul 06 17:18:03.925952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925913 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" Jul 06 17:18:03.925952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925920 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" Jul 06 17:18:03.925952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925925 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" Jul 06 17:18:03.926229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925977 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" Jul 06 17:18:03.926229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925984 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="59857777-bc08-47f7-98e4-8c4c50332159" containerName="maas-api" Jul 06 17:18:03.926229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.925989 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2466a4a-c778-4152-b02b-cf3a4647baee" containerName="maas-api" Jul 06 17:18:03.928867 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.928849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:03.931801 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.931784 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-2cd1eb-openshift-default-dockercfg-w54wd\"" Jul 06 17:18:03.947419 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.947386 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx"] Jul 06 17:18:03.956621 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:03.956591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:04.033122 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-envoy\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.033122 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-data\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.033337 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033256 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-podinfo\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.033337 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033304 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-certs\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.033428 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-credential-socket\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.033493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istiod-ca-cert\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.033543 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-socket\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.033543 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-token\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.033659 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.033574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9hbv\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-kube-api-access-q9hbv\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134286 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-podinfo\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134286 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-certs\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134549 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-credential-socket\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134549 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istiod-ca-cert\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134549 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-socket\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134549 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-token\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134549 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9hbv\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-kube-api-access-q9hbv\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134549 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-envoy\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134549 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-data\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134768 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-credential-socket\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134883 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-data\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.134952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.134905 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-certs\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.135113 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.135077 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-socket\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.135298 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.135273 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istiod-ca-cert\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.136764 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.136742 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-envoy\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.136975 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.136956 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-podinfo\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.143645 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.143620 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-token\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.143764 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.143746 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9hbv\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-kube-api-access-q9hbv\") pod \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:04.234601 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.234544 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:04.239612 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:04.239593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:06.826985 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:06.826952 2574 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 06 17:18:06.826985 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424" Netns:"/var/run/netns/cb27f5cc-e169-4c2d-89f5-9d214f87a886" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:06.826985 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:06.826985 ip-10-0-130-239 kubenswrapper[2574]: > Jul 06 17:18:06.827373 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:06.827011 2574 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 06 17:18:06.827373 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424" Netns:"/var/run/netns/cb27f5cc-e169-4c2d-89f5-9d214f87a886" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:06.827373 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:06.827373 ip-10-0-130-239 kubenswrapper[2574]: > pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:06.827373 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:06.827028 2574 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 06 17:18:06.827373 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424" Netns:"/var/run/netns/cb27f5cc-e169-4c2d-89f5-9d214f87a886" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:06.827373 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:06.827373 ip-10-0-130-239 kubenswrapper[2574]: > pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:06.827665 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:06.827112 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress(dfe27f9c-0f47-4c29-ac62-6da6d02f94a4)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress(dfe27f9c-0f47-4c29-ac62-6da6d02f94a4)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424\\\" Netns:\\\"/var/run/netns/cb27f5cc-e169-4c2d-89f5-9d214f87a886\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=a1181b4051817d6775e9073134282333ebfd6ce1eb60824192d88f4521f97424;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get \\\"https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Jul 06 17:18:07.369743 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:07.369709 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:07.370030 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:07.370019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:08.340148 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.340123 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:18:08.375510 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375403 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-certs\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.375510 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375445 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c00122-e6d5-43e5-8869-12def26a6934-istiod-ca-cert\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.375510 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375497 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-socket\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.375789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375524 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc8bs\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-kube-api-access-hc8bs\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.375789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375550 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c00122-e6d5-43e5-8869-12def26a6934-istio-podinfo\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.375789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375592 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-istio-token\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.375789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375644 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-envoy\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.375789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375699 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-data\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.375789 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375781 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-credential-socket\") pod \"b1c00122-e6d5-43e5-8869-12def26a6934\" (UID: \"b1c00122-e6d5-43e5-8869-12def26a6934\") " Jul 06 17:18:08.376089 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375839 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c00122-e6d5-43e5-8869-12def26a6934-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:18:08.376089 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376020 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:08.376089 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376044 2574 generic.go:356] "Generic (PLEG): container finished" podID="b1c00122-e6d5-43e5-8869-12def26a6934" containerID="f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a" exitCode=0 Jul 06 17:18:08.376237 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376113 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" Jul 06 17:18:08.376237 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376162 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" event={"ID":"b1c00122-e6d5-43e5-8869-12def26a6934","Type":"ContainerDied","Data":"f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a"} Jul 06 17:18:08.376237 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376203 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.376237 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376212 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc" event={"ID":"b1c00122-e6d5-43e5-8869-12def26a6934","Type":"ContainerDied","Data":"b421fc7af8f22521bdb83419b5753c17c6f3900c6c4fe71810d0156f0b173ae5"} Jul 06 17:18:08.376237 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376221 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c00122-e6d5-43e5-8869-12def26a6934-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.376237 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.375841 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:08.376560 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376243 2574 scope.go:117] "RemoveContainer" containerID="f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a" Jul 06 17:18:08.376560 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.376403 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:08.377067 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.377032 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-data" (OuterVolumeSpecName: "istio-data") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:08.378386 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.378357 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1c00122-e6d5-43e5-8869-12def26a6934-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:18:08.378386 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.378376 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-kube-api-access-hc8bs" (OuterVolumeSpecName: "kube-api-access-hc8bs") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "kube-api-access-hc8bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:18:08.378571 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.378553 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:18:08.378617 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.378606 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-istio-token" (OuterVolumeSpecName: "istio-token") pod "b1c00122-e6d5-43e5-8869-12def26a6934" (UID: "b1c00122-e6d5-43e5-8869-12def26a6934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:18:08.399598 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.399569 2574 scope.go:117] "RemoveContainer" containerID="7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be" Jul 06 17:18:08.407556 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.407534 2574 scope.go:117] "RemoveContainer" containerID="f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a" Jul 06 17:18:08.407851 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:08.407829 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a\": container with ID starting with f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a not found: ID does not exist" containerID="f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a" Jul 06 17:18:08.407903 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.407860 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a"} err="failed to get container status \"f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a\": rpc error: code = NotFound desc = could not find container \"f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a\": container with ID starting with f3ddbbd67594076091d9f6376230319352cd728d3b32951edb983fbf13fad49a not found: ID does not exist" Jul 06 17:18:08.407903 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.407884 2574 scope.go:117] "RemoveContainer" containerID="7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be" Jul 06 17:18:08.408121 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:08.408103 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be\": container with ID starting with 7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be not found: ID does not exist" containerID="7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be" Jul 06 17:18:08.408186 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.408131 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be"} err="failed to get container status \"7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be\": rpc error: code = NotFound desc = could not find container \"7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be\": container with ID starting with 7f03811e2a9448288be3598456fd1fd149032f272f25c69f6fa50f6e139635be not found: ID does not exist" Jul 06 17:18:08.476905 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.476868 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.476905 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.476896 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.476905 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.476911 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.477136 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.476924 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8bs\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-kube-api-access-hc8bs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.477136 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.476937 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c00122-e6d5-43e5-8869-12def26a6934-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.477136 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.476948 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c00122-e6d5-43e5-8869-12def26a6934-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.477136 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.476958 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c00122-e6d5-43e5-8869-12def26a6934-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:18:08.567742 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.567705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:08.638390 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.638312 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:08.719106 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.719073 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc"] Jul 06 17:18:08.723656 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:08.723630 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9c929f-openshift-default-d56778ccc-wk8lc"] Jul 06 17:18:09.414176 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:09.414144 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:09.748041 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:09.748012 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" path="/var/lib/kubelet/pods/b1c00122-e6d5-43e5-8869-12def26a6934/volumes" Jul 06 17:18:09.961788 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:09.961750 2574 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 06 17:18:09.961788 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d" Netns:"/var/run/netns/7449f5b9-53db-43cb-a2ac-99bf12862ba5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:09.961788 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:09.961788 ip-10-0-130-239 kubenswrapper[2574]: > Jul 06 17:18:09.961998 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:09.961805 2574 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 06 17:18:09.961998 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d" Netns:"/var/run/netns/7449f5b9-53db-43cb-a2ac-99bf12862ba5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:09.961998 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:09.961998 ip-10-0-130-239 kubenswrapper[2574]: > pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:09.961998 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:09.961826 2574 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 06 17:18:09.961998 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d" Netns:"/var/run/netns/7449f5b9-53db-43cb-a2ac-99bf12862ba5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:09.961998 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:09.961998 ip-10-0-130-239 kubenswrapper[2574]: > pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:09.962238 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:09.961909 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress(dfe27f9c-0f47-4c29-ac62-6da6d02f94a4)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress(dfe27f9c-0f47-4c29-ac62-6da6d02f94a4)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d\\\" Netns:\\\"/var/run/netns/7449f5b9-53db-43cb-a2ac-99bf12862ba5\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=6d48e99f551bc43a0d9f24ff59b7269a3f838086beb432c71669cd7527de300d;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get \\\"https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Jul 06 17:18:10.270012 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:10.269981 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:21.520360 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:21.520316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:22.742402 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:22.742354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:22.742865 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:22.742851 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:25.332714 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:25.332680 2574 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 06 17:18:25.332714 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702" Netns:"/var/run/netns/e7e861a7-2208-45a1-9461-ad413a8e915b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:25.332714 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:25.332714 ip-10-0-130-239 kubenswrapper[2574]: > Jul 06 17:18:25.333135 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:25.332739 2574 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 06 17:18:25.333135 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702" Netns:"/var/run/netns/e7e861a7-2208-45a1-9461-ad413a8e915b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:25.333135 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:25.333135 ip-10-0-130-239 kubenswrapper[2574]: > pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:25.333135 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:25.332758 2574 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 06 17:18:25.333135 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702" Netns:"/var/run/netns/e7e861a7-2208-45a1-9461-ad413a8e915b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:25.333135 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:25.333135 ip-10-0-130-239 kubenswrapper[2574]: > pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:25.333377 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:25.332839 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress(dfe27f9c-0f47-4c29-ac62-6da6d02f94a4)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress(dfe27f9c-0f47-4c29-ac62-6da6d02f94a4)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702\\\" Netns:\\\"/var/run/netns/e7e861a7-2208-45a1-9461-ad413a8e915b\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=1230ac767c904f889235966d200c8ff2287a4dcefd026d508f3e2aa5d5740702;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get \\\"https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Jul 06 17:18:28.598844 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:28.598809 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:31.668547 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:31.668502 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:31.877872 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:31.877835 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:37.742635 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:37.742592 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:37.743068 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:37.743051 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:40.334713 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:40.334684 2574 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 06 17:18:40.334713 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab" Netns:"/var/run/netns/afc8ba65-227a-4f38-8d71-ed60df7f9d16" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:40.334713 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:40.334713 ip-10-0-130-239 kubenswrapper[2574]: > Jul 06 17:18:40.335122 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:40.334735 2574 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 06 17:18:40.335122 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab" Netns:"/var/run/netns/afc8ba65-227a-4f38-8d71-ed60df7f9d16" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:40.335122 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:40.335122 ip-10-0-130-239 kubenswrapper[2574]: > pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:40.335122 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:40.334755 2574 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 06 17:18:40.335122 ip-10-0-130-239 kubenswrapper[2574]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab" Netns:"/var/run/netns/afc8ba65-227a-4f38-8d71-ed60df7f9d16" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get "https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s": context deadline exceeded Jul 06 17:18:40.335122 ip-10-0-130-239 kubenswrapper[2574]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 17:18:40.335122 ip-10-0-130-239 kubenswrapper[2574]: > pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:40.335358 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:40.334825 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress(dfe27f9c-0f47-4c29-ac62-6da6d02f94a4)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress(dfe27f9c-0f47-4c29-ac62-6da6d02f94a4)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx_openshift-ingress_dfe27f9c-0f47-4c29-ac62-6da6d02f94a4_0(6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab): error adding pod openshift-ingress_e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab\\\" Netns:\\\"/var/run/netns/afc8ba65-227a-4f38-8d71-ed60df7f9d16\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx;K8S_POD_INFRA_CONTAINER_ID=6d4eacd1b888f3a978afa690b50619ad8c8eb164420f84a078835cc9152e28ab;K8S_POD_UID=dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx] networking: Multus: [openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4]: error waiting for pod: Get \\\"https://a63e3a650d7d6496d8a9d7a34922820d-ee5eddcae3b91cb3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" Jul 06 17:18:53.745028 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:53.744983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:53.745579 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:53.745450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:53.888799 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:53.888773 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx"] Jul 06 17:18:53.890284 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:18:53.890249 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe27f9c_0f47_4c29_ac62_6da6d02f94a4.slice/crio-673c8116e5f81bec4b07dc371b28410305ac2394b65626953d001434420a7021 WatchSource:0}: Error finding container 673c8116e5f81bec4b07dc371b28410305ac2394b65626953d001434420a7021: Status 404 returned error can't find the container with id 673c8116e5f81bec4b07dc371b28410305ac2394b65626953d001434420a7021 Jul 06 17:18:53.892359 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:53.892329 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:18:53.892422 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:53.892395 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:18:53.892475 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:53.892426 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:18:54.599106 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:54.599071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" event={"ID":"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4","Type":"ContainerStarted","Data":"a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98"} Jul 06 17:18:54.599290 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:54.599123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" event={"ID":"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4","Type":"ContainerStarted","Data":"673c8116e5f81bec4b07dc371b28410305ac2394b65626953d001434420a7021"} Jul 06 17:18:54.635957 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:54.635910 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podStartSLOduration=51.635893493 podStartE2EDuration="51.635893493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:18:54.631725564 +0000 UTC m=+2431.401617104" watchObservedRunningTime="2026-07-06 17:18:54.635893493 +0000 UTC m=+2431.405785033" Jul 06 17:18:55.240208 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:55.240169 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:55.245006 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:55.244979 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:55.603491 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:55.603400 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:55.604386 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:55.604365 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:18:56.902018 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:56.901987 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:57.011386 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.011350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:57.116044 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.116011 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh"] Jul 06 17:18:57.116411 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.116398 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" Jul 06 17:18:57.116469 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.116412 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" Jul 06 17:18:57.116469 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.116421 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" Jul 06 17:18:57.116469 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.116427 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" Jul 06 17:18:57.116580 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.116515 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db4afad-b278-4ec5-9ff8-2095f8d7fed1" containerName="istio-proxy" Jul 06 17:18:57.116580 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.116523 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" Jul 06 17:18:57.119596 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.119578 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.122845 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.122825 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-2cd1eb-serving-cert\"" Jul 06 17:18:57.146052 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.146025 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh"] Jul 06 17:18:57.204183 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.204103 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:57.221401 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.221356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqwh\" (UniqueName: \"kubernetes.io/projected/528f3502-4137-462c-87e3-1f7b373acab2-kube-api-access-mgqwh\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.221586 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.221494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.221586 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.221545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f3502-4137-462c-87e3-1f7b373acab2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.323043 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.323006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqwh\" (UniqueName: \"kubernetes.io/projected/528f3502-4137-462c-87e3-1f7b373acab2-kube-api-access-mgqwh\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.323221 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.323063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.323221 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.323091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f3502-4137-462c-87e3-1f7b373acab2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.323221 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:57.323211 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-2cd1eb-serving-cert: secret "maas-api-e2e-shared-a-2cd1eb-serving-cert" not found Jul 06 17:18:57.323343 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:18:57.323267 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls podName:528f3502-4137-462c-87e3-1f7b373acab2 nodeName:}" failed. No retries permitted until 2026-07-06 17:18:57.823252849 +0000 UTC m=+2434.593144374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls") pod "maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" (UID: "528f3502-4137-462c-87e3-1f7b373acab2") : secret "maas-api-e2e-shared-a-2cd1eb-serving-cert" not found Jul 06 17:18:57.323724 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.323707 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f3502-4137-462c-87e3-1f7b373acab2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.346609 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.346576 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqwh\" (UniqueName: \"kubernetes.io/projected/528f3502-4137-462c-87e3-1f7b373acab2-kube-api-access-mgqwh\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.510399 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.510362 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:57.828188 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.828107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.830435 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.830412 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls\") pod \"maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:57.932660 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:57.932626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:58.029332 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:58.029298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:58.030342 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:58.030316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:18:58.179480 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:58.178904 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh"] Jul 06 17:18:58.619054 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:58.619011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" event={"ID":"528f3502-4137-462c-87e3-1f7b373acab2","Type":"ContainerStarted","Data":"98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57"} Jul 06 17:18:58.619221 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:58.619165 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:18:58.619221 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:58.619211 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" event={"ID":"528f3502-4137-462c-87e3-1f7b373acab2","Type":"ContainerStarted","Data":"a9d5972d294a7d62f650d42baf2008eaae7cf4ebc6caae7e6c55f8329386dad6"} Jul 06 17:18:58.640441 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:18:58.640393 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" podStartSLOduration=1.299790875 podStartE2EDuration="1.640372924s" totalImagesPullingTime="340.582049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:18:58.638624537 +0000 UTC m=+2435.408516079" watchObservedRunningTime="2026-07-06 17:18:58.640372924 +0000 UTC m=+2435.410264465" Jul 06 17:19:01.288970 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:01.288925 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:01.685881 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:01.685771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:01.970369 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:01.970278 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x"] Jul 06 17:19:01.970888 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:01.970867 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c00122-e6d5-43e5-8869-12def26a6934" containerName="istio-proxy" Jul 06 17:19:01.973796 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:01.973774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:01.976845 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:01.976824 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-0ebdd9-openshift-default-dockercfg-nvmv7\"" Jul 06 17:19:01.983423 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:01.983395 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:01.992180 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:01.992156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x"] Jul 06 17:19:02.067229 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-certs\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.067422 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-data\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.067422 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knpb8\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-kube-api-access-knpb8\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.067422 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-credential-socket\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.067422 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87b9408-8c39-498b-906b-f664cea0d922-istiod-ca-cert\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.067661 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-istio-token\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.067661 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-envoy\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.067661 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87b9408-8c39-498b-906b-f664cea0d922-istio-podinfo\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.067661 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.067578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-socket\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168418 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-certs\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168636 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-data\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168636 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knpb8\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-kube-api-access-knpb8\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168636 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-credential-socket\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168636 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87b9408-8c39-498b-906b-f664cea0d922-istiod-ca-cert\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168636 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-istio-token\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168904 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-envoy\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168904 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168859 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-data\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.168904 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168860 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-certs\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.169056 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87b9408-8c39-498b-906b-f664cea0d922-istio-podinfo\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.169056 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.168966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-socket\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.169299 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.169275 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87b9408-8c39-498b-906b-f664cea0d922-istiod-ca-cert\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.169356 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.169306 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-socket\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.169420 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.169404 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-credential-socket\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.171493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.171444 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-envoy\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.171793 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.171771 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87b9408-8c39-498b-906b-f664cea0d922-istio-podinfo\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.177120 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.177089 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-istio-token\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.177227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.177209 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knpb8\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-kube-api-access-knpb8\") pod \"e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.284413 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.284378 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:02.388093 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.388065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:02.456190 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.456158 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x"] Jul 06 17:19:02.458944 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:19:02.458905 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87b9408_8c39_498b_906b_f664cea0d922.slice/crio-025218c2b7f6dc957dd5a7fc51fedaaaa07192c15561fec290a23756cfec0fbc WatchSource:0}: Error finding container 025218c2b7f6dc957dd5a7fc51fedaaaa07192c15561fec290a23756cfec0fbc: Status 404 returned error can't find the container with id 025218c2b7f6dc957dd5a7fc51fedaaaa07192c15561fec290a23756cfec0fbc Jul 06 17:19:02.461167 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.461136 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:02.461257 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.461211 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:02.461320 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.461258 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:02.646538 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.646051 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" event={"ID":"a87b9408-8c39-498b-906b-f664cea0d922","Type":"ContainerStarted","Data":"0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215"} Jul 06 17:19:02.646538 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.646122 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" event={"ID":"a87b9408-8c39-498b-906b-f664cea0d922","Type":"ContainerStarted","Data":"025218c2b7f6dc957dd5a7fc51fedaaaa07192c15561fec290a23756cfec0fbc"} Jul 06 17:19:02.705914 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:02.705846 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" podStartSLOduration=1.70582345 podStartE2EDuration="1.70582345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:19:02.703166081 +0000 UTC m=+2439.473057621" watchObservedRunningTime="2026-07-06 17:19:02.70582345 +0000 UTC m=+2439.475714991" Jul 06 17:19:03.285619 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:03.285577 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:03.290722 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:03.290690 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:03.653701 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:03.653606 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:03.654757 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:03.654733 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:04.640932 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:04.640887 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:19:04.867788 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:04.867758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:04.966206 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:04.966120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:05.097485 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.097431 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn"] Jul 06 17:19:05.101317 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.101294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.103776 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.103750 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-0ebdd9-serving-cert\"" Jul 06 17:19:05.113952 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.113929 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn"] Jul 06 17:19:05.155503 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.154120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:05.199513 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.199474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50926d06-69d8-44a1-a937-6bd435d74726-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.199684 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.199546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50926d06-69d8-44a1-a937-6bd435d74726-maas-api-tls\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.199684 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.199607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sg87\" (UniqueName: \"kubernetes.io/projected/50926d06-69d8-44a1-a937-6bd435d74726-kube-api-access-5sg87\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.252839 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.252803 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:05.301164 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.301123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sg87\" (UniqueName: \"kubernetes.io/projected/50926d06-69d8-44a1-a937-6bd435d74726-kube-api-access-5sg87\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.301341 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.301224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50926d06-69d8-44a1-a937-6bd435d74726-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.301341 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.301266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50926d06-69d8-44a1-a937-6bd435d74726-maas-api-tls\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.302014 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.301975 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50926d06-69d8-44a1-a937-6bd435d74726-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.303867 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.303837 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50926d06-69d8-44a1-a937-6bd435d74726-maas-api-tls\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.313087 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.313060 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sg87\" (UniqueName: \"kubernetes.io/projected/50926d06-69d8-44a1-a937-6bd435d74726-kube-api-access-5sg87\") pod \"maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.411867 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.411835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:05.584388 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.584356 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn"] Jul 06 17:19:05.587183 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:19:05.587152 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50926d06_69d8_44a1_a937_6bd435d74726.slice/crio-3bdbb6c60e1261806b0493d30bce09cd489595d65cacbbe439c926bcde9f1e18 WatchSource:0}: Error finding container 3bdbb6c60e1261806b0493d30bce09cd489595d65cacbbe439c926bcde9f1e18: Status 404 returned error can't find the container with id 3bdbb6c60e1261806b0493d30bce09cd489595d65cacbbe439c926bcde9f1e18 Jul 06 17:19:05.664374 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.664338 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" event={"ID":"50926d06-69d8-44a1-a937-6bd435d74726","Type":"ContainerStarted","Data":"3bdbb6c60e1261806b0493d30bce09cd489595d65cacbbe439c926bcde9f1e18"} Jul 06 17:19:05.665853 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.665804 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:05.765428 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:05.765403 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:06.670220 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:06.670184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" event={"ID":"50926d06-69d8-44a1-a937-6bd435d74726","Type":"ContainerStarted","Data":"1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a"} Jul 06 17:19:06.670750 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:06.670326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:06.693745 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:06.693691 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" podStartSLOduration=1.265350208 podStartE2EDuration="1.693672321s" totalImagesPullingTime="428.322113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:19:06.691165014 +0000 UTC m=+2443.461056568" watchObservedRunningTime="2026-07-06 17:19:06.693672321 +0000 UTC m=+2443.463563862" Jul 06 17:19:08.681885 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:08.681853 2574 generic.go:356] "Generic (PLEG): container finished" podID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerID="a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98" exitCode=0 Jul 06 17:19:08.682303 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:08.681930 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" event={"ID":"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4","Type":"ContainerDied","Data":"a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98"} Jul 06 17:19:08.682432 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:08.682416 2574 scope.go:117] "RemoveContainer" containerID="a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98" Jul 06 17:19:08.683563 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:08.683537 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:08.683627 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:08.683600 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:08.683664 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:08.683631 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:09.315227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:09.315195 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:09.688526 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:09.688425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" event={"ID":"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4","Type":"ContainerStarted","Data":"b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d"} Jul 06 17:19:10.240069 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:10.240016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:19:10.241416 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:10.241381 2574 patch_prober.go:28] interesting pod/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 17:19:10.241585 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:10.241467 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 17:19:10.604139 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:10.604046 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:19:11.240848 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:11.240365 2574 patch_prober.go:28] interesting pod/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 17:19:11.240848 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:11.240434 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 17:19:12.241020 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:12.240980 2574 patch_prober.go:28] interesting pod/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 17:19:12.241579 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:12.241050 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 17:19:12.681369 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:12.681288 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:19:13.240565 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:13.240061 2574 patch_prober.go:28] interesting pod/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 17:19:13.240565 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:13.240127 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 17:19:14.251049 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:14.251016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:19:14.253232 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:14.253190 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:19:18.771566 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:18.771529 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:18.878066 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:18.878028 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:19.695666 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:19.695623 2574 patch_prober.go:28] interesting pod/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 17:19:19.695852 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:19.695702 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 17:19:20.745762 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:20.745728 2574 generic.go:356] "Generic (PLEG): container finished" podID="a87b9408-8c39-498b-906b-f664cea0d922" containerID="0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215" exitCode=0 Jul 06 17:19:20.745762 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:20.745765 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" event={"ID":"a87b9408-8c39-498b-906b-f664cea0d922","Type":"ContainerDied","Data":"0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215"} Jul 06 17:19:20.746286 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:20.746130 2574 scope.go:117] "RemoveContainer" containerID="0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215" Jul 06 17:19:20.747188 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:20.747160 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:20.747242 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:20.747228 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:20.747277 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:20.747258 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 17:19:21.759919 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:21.759886 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" event={"ID":"a87b9408-8c39-498b-906b-f664cea0d922","Type":"ContainerStarted","Data":"729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153"} Jul 06 17:19:22.285114 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:22.285080 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:22.285339 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:22.285314 2574 patch_prober.go:28] interesting pod/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 17:19:22.285433 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:22.285391 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 17:19:23.285672 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:23.285641 2574 patch_prober.go:28] interesting pod/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 17:19:23.286043 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:23.285694 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 17:19:24.284826 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:24.284789 2574 patch_prober.go:28] interesting pod/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 17:19:24.285038 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:24.284888 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 17:19:25.284902 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:25.284869 2574 patch_prober.go:28] interesting pod/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 17:19:25.285323 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:25.284923 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 17:19:26.288185 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:26.288153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:26.288695 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:26.288557 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:26.289309 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:26.289286 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:19:26.376787 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:26.376740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:26.476861 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:26.476822 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:19:30.970649 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:19:30.970615 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:20:16.488975 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:20:16.488893 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:22:07.292786 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:22:07.292750 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:22:14.889892 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:22:14.889854 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:22:15.881496 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:22:15.881435 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:22:17.577787 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:22:17.577750 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:22:41.385630 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:22:41.385599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:22:42.330090 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:22:42.330057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:22:42.426118 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:22:42.426079 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:23:01.890977 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:23:01.890932 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:23:31.046156 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:23:31.046125 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:23:45.589406 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:23:45.589370 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:23:46.506319 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:23:46.506283 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:23:54.829960 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:23:54.829926 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:24:19.441485 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:24:19.441432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:24:19.552425 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:24:19.552387 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:24:22.687854 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:24:22.687817 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:24:27.211102 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:24:27.211070 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:24:27.357441 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:24:27.357388 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:00.996998 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:00.996967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:02.003068 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:02.003031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:03.089308 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:03.089276 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:03.527607 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:03.527577 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:03.711791 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:03.711752 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:03.815756 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:03.815672 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:04.001025 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.000991 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh"] Jul 06 17:25:04.001274 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.001248 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" podUID="528f3502-4137-462c-87e3-1f7b373acab2" containerName="maas-api" containerID="cri-o://98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57" gracePeriod=30 Jul 06 17:25:04.199994 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.199952 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:04.250723 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.250699 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:25:04.315321 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.315290 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f3502-4137-462c-87e3-1f7b373acab2-openshift-service-ca\") pod \"528f3502-4137-462c-87e3-1f7b373acab2\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " Jul 06 17:25:04.315321 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.315325 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgqwh\" (UniqueName: \"kubernetes.io/projected/528f3502-4137-462c-87e3-1f7b373acab2-kube-api-access-mgqwh\") pod \"528f3502-4137-462c-87e3-1f7b373acab2\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " Jul 06 17:25:04.315644 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.315361 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls\") pod \"528f3502-4137-462c-87e3-1f7b373acab2\" (UID: \"528f3502-4137-462c-87e3-1f7b373acab2\") " Jul 06 17:25:04.315785 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.315753 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528f3502-4137-462c-87e3-1f7b373acab2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "528f3502-4137-462c-87e3-1f7b373acab2" (UID: "528f3502-4137-462c-87e3-1f7b373acab2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:25:04.317569 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.317536 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f3502-4137-462c-87e3-1f7b373acab2-kube-api-access-mgqwh" (OuterVolumeSpecName: "kube-api-access-mgqwh") pod "528f3502-4137-462c-87e3-1f7b373acab2" (UID: "528f3502-4137-462c-87e3-1f7b373acab2"). InnerVolumeSpecName "kube-api-access-mgqwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:25:04.317669 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.317581 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "528f3502-4137-462c-87e3-1f7b373acab2" (UID: "528f3502-4137-462c-87e3-1f7b373acab2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:25:04.388874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.388776 2574 generic.go:356] "Generic (PLEG): container finished" podID="528f3502-4137-462c-87e3-1f7b373acab2" containerID="98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57" exitCode=0 Jul 06 17:25:04.388874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.388818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" event={"ID":"528f3502-4137-462c-87e3-1f7b373acab2","Type":"ContainerDied","Data":"98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57"} Jul 06 17:25:04.388874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.388823 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" Jul 06 17:25:04.388874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.388868 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh" event={"ID":"528f3502-4137-462c-87e3-1f7b373acab2","Type":"ContainerDied","Data":"a9d5972d294a7d62f650d42baf2008eaae7cf4ebc6caae7e6c55f8329386dad6"} Jul 06 17:25:04.389202 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.388892 2574 scope.go:117] "RemoveContainer" containerID="98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57" Jul 06 17:25:04.399838 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.399817 2574 scope.go:117] "RemoveContainer" containerID="98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57" Jul 06 17:25:04.400366 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:25:04.400343 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57\": container with ID starting with 98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57 not found: ID does not exist" containerID="98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57" Jul 06 17:25:04.400439 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.400379 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57"} err="failed to get container status \"98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57\": rpc error: code = NotFound desc = could not find container \"98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57\": container with ID starting with 98444a887db1863d0ec2bc6940398c9b4278c08f9a65559160be659d70bb6d57 not found: ID does not exist" Jul 06 17:25:04.416142 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.416108 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f3502-4137-462c-87e3-1f7b373acab2-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:04.416142 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.416135 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgqwh\" (UniqueName: \"kubernetes.io/projected/528f3502-4137-462c-87e3-1f7b373acab2-kube-api-access-mgqwh\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:04.416295 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.416151 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f3502-4137-462c-87e3-1f7b373acab2-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:04.425273 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.425246 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh"] Jul 06 17:25:04.429291 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:04.429270 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2cd1eb-57c6ff4db7-lqhlh"] Jul 06 17:25:05.748507 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:05.748473 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="528f3502-4137-462c-87e3-1f7b373acab2" path="/var/lib/kubelet/pods/528f3502-4137-462c-87e3-1f7b373acab2/volumes" Jul 06 17:25:07.062646 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:07.062615 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:07.173000 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:07.172967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:08.630805 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:08.630769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:29.933908 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:29.933873 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:29.994061 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:29.994018 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:46.065307 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:46.065270 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:46.314956 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:46.314924 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx"] Jul 06 17:25:46.315140 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:46.315118 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" containerID="cri-o://b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d" gracePeriod=30 Jul 06 17:25:46.324327 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:46.324299 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:46.552040 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:46.552012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:46.878439 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:46.878396 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:47.295144 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:47.295112 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:47.490507 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:47.490436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:47.590868 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:47.590779 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:47.731266 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:47.731228 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn"] Jul 06 17:25:47.731529 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:47.731496 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" podUID="50926d06-69d8-44a1-a937-6bd435d74726" containerName="maas-api" containerID="cri-o://1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a" gracePeriod=30 Jul 06 17:25:47.767742 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:47.767706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:47.977241 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:47.977213 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:25:48.144548 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.144435 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50926d06-69d8-44a1-a937-6bd435d74726-openshift-service-ca\") pod \"50926d06-69d8-44a1-a937-6bd435d74726\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " Jul 06 17:25:48.144548 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.144529 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sg87\" (UniqueName: \"kubernetes.io/projected/50926d06-69d8-44a1-a937-6bd435d74726-kube-api-access-5sg87\") pod \"50926d06-69d8-44a1-a937-6bd435d74726\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " Jul 06 17:25:48.144787 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.144559 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50926d06-69d8-44a1-a937-6bd435d74726-maas-api-tls\") pod \"50926d06-69d8-44a1-a937-6bd435d74726\" (UID: \"50926d06-69d8-44a1-a937-6bd435d74726\") " Jul 06 17:25:48.144851 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.144805 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50926d06-69d8-44a1-a937-6bd435d74726-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50926d06-69d8-44a1-a937-6bd435d74726" (UID: "50926d06-69d8-44a1-a937-6bd435d74726"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:25:48.146653 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.146629 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50926d06-69d8-44a1-a937-6bd435d74726-kube-api-access-5sg87" (OuterVolumeSpecName: "kube-api-access-5sg87") pod "50926d06-69d8-44a1-a937-6bd435d74726" (UID: "50926d06-69d8-44a1-a937-6bd435d74726"). InnerVolumeSpecName "kube-api-access-5sg87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:25:48.146759 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.146720 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50926d06-69d8-44a1-a937-6bd435d74726-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50926d06-69d8-44a1-a937-6bd435d74726" (UID: "50926d06-69d8-44a1-a937-6bd435d74726"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:25:48.245495 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.245421 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sg87\" (UniqueName: \"kubernetes.io/projected/50926d06-69d8-44a1-a937-6bd435d74726-kube-api-access-5sg87\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:48.245495 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.245492 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50926d06-69d8-44a1-a937-6bd435d74726-maas-api-tls\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:48.245705 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.245508 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50926d06-69d8-44a1-a937-6bd435d74726-openshift-service-ca\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:48.593479 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.593422 2574 generic.go:356] "Generic (PLEG): container finished" podID="50926d06-69d8-44a1-a937-6bd435d74726" containerID="1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a" exitCode=0 Jul 06 17:25:48.593930 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.593480 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" event={"ID":"50926d06-69d8-44a1-a937-6bd435d74726","Type":"ContainerDied","Data":"1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a"} Jul 06 17:25:48.593930 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.593507 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" Jul 06 17:25:48.593930 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.593589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn" event={"ID":"50926d06-69d8-44a1-a937-6bd435d74726","Type":"ContainerDied","Data":"3bdbb6c60e1261806b0493d30bce09cd489595d65cacbbe439c926bcde9f1e18"} Jul 06 17:25:48.593930 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.593619 2574 scope.go:117] "RemoveContainer" containerID="1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a" Jul 06 17:25:48.602787 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.602770 2574 scope.go:117] "RemoveContainer" containerID="1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a" Jul 06 17:25:48.603049 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:25:48.603031 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a\": container with ID starting with 1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a not found: ID does not exist" containerID="1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a" Jul 06 17:25:48.603097 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.603058 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a"} err="failed to get container status \"1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a\": rpc error: code = NotFound desc = could not find container \"1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a\": container with ID starting with 1eb639b89acfc7f8bb9353c0cff93c803bbd4704fc3e46b8601838398b5fc01a not found: ID does not exist" Jul 06 17:25:48.619905 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.619864 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn"] Jul 06 17:25:48.622257 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:48.622233 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0ebdd9-fcbc4dc97-pwswn"] Jul 06 17:25:49.748319 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:49.748285 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50926d06-69d8-44a1-a937-6bd435d74726" path="/var/lib/kubelet/pods/50926d06-69d8-44a1-a937-6bd435d74726/volumes" Jul 06 17:25:51.575724 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.575669 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:25:51.610479 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.610424 2574 generic.go:356] "Generic (PLEG): container finished" podID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerID="b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d" exitCode=0 Jul 06 17:25:51.610650 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.610491 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" Jul 06 17:25:51.610650 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.610499 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" event={"ID":"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4","Type":"ContainerDied","Data":"b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d"} Jul 06 17:25:51.610650 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.610548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx" event={"ID":"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4","Type":"ContainerDied","Data":"673c8116e5f81bec4b07dc371b28410305ac2394b65626953d001434420a7021"} Jul 06 17:25:51.610650 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.610575 2574 scope.go:117] "RemoveContainer" containerID="b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d" Jul 06 17:25:51.622421 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.622395 2574 scope.go:117] "RemoveContainer" containerID="a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98" Jul 06 17:25:51.630545 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.630526 2574 scope.go:117] "RemoveContainer" containerID="b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d" Jul 06 17:25:51.630873 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:25:51.630850 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d\": container with ID starting with b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d not found: ID does not exist" containerID="b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d" Jul 06 17:25:51.630974 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.630880 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d"} err="failed to get container status \"b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d\": rpc error: code = NotFound desc = could not find container \"b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d\": container with ID starting with b3a2fb8ea8fc6c9ffa58c5629c2480362f7ae9bc26f31e701dfed366fb4b3e2d not found: ID does not exist" Jul 06 17:25:51.630974 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.630904 2574 scope.go:117] "RemoveContainer" containerID="a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98" Jul 06 17:25:51.631204 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:25:51.631179 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98\": container with ID starting with a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98 not found: ID does not exist" containerID="a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98" Jul 06 17:25:51.631250 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.631214 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98"} err="failed to get container status \"a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98\": rpc error: code = NotFound desc = could not find container \"a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98\": container with ID starting with a258ed471f987dc04653cdf9848db58c27a61947904ead4d51ef1b95078ffa98 not found: ID does not exist" Jul 06 17:25:51.677110 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677075 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istiod-ca-cert\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677281 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677119 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-credential-socket\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677281 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677141 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9hbv\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-kube-api-access-q9hbv\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677281 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677192 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-socket\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677281 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677229 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-data\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677281 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677252 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-certs\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677601 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677282 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-envoy\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677601 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677326 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-podinfo\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677601 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677434 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-token\") pod \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\" (UID: \"dfe27f9c-0f47-4c29-ac62-6da6d02f94a4\") " Jul 06 17:25:51.677601 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677470 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:51.677804 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677585 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:25:51.677804 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677645 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:51.677927 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677819 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:51.677927 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677828 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.677927 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677864 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.677927 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.677881 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.678422 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.678396 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:51.679597 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.679563 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:51.679689 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.679630 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-kube-api-access-q9hbv" (OuterVolumeSpecName: "kube-api-access-q9hbv") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "kube-api-access-q9hbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:25:51.679689 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.679648 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:25:51.679689 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.679651 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" (UID: "dfe27f9c-0f47-4c29-ac62-6da6d02f94a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:25:51.778596 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.778570 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.778596 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.778594 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.778596 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.778603 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.778827 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.778623 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9hbv\" (UniqueName: \"kubernetes.io/projected/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-kube-api-access-q9hbv\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.778827 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.778633 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.778827 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.778640 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:51.927581 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.927498 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx"] Jul 06 17:25:51.932003 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:51.931970 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-2cd1eb-openshift-default-6bc4c5c9d7-7pxmx"] Jul 06 17:25:53.748639 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:53.748596 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" path="/var/lib/kubelet/pods/dfe27f9c-0f47-4c29-ac62-6da6d02f94a4/volumes" Jul 06 17:25:53.857361 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:53.857315 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:53.959181 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:53.959137 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:54.192088 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:54.192003 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x"] Jul 06 17:25:54.192256 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:54.192222 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" containerID="cri-o://729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153" gracePeriod=30 Jul 06 17:25:54.197053 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:54.197028 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:54.427906 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:54.427858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:25:56.291383 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:56.291344 2574 patch_prober.go:28] interesting pod/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 17:25:56.291851 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:56.291418 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 17:25:59.453552 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.453493 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:25:59.551313 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551276 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-data\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551331 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-certs\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551377 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-envoy\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551409 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-istio-token\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551493 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-socket\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551668 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551498 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knpb8\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-kube-api-access-knpb8\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551668 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551533 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-credential-socket\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551668 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551560 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87b9408-8c39-498b-906b-f664cea0d922-istio-podinfo\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551668 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551621 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87b9408-8c39-498b-906b-f664cea0d922-istiod-ca-cert\") pod \"a87b9408-8c39-498b-906b-f664cea0d922\" (UID: \"a87b9408-8c39-498b-906b-f664cea0d922\") " Jul 06 17:25:59.551668 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551644 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:59.551861 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551776 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:59.551955 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551939 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-certs\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.552010 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.551959 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-credential-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.552073 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.552053 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87b9408-8c39-498b-906b-f664cea0d922-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:25:59.552135 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.552110 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-data" (OuterVolumeSpecName: "istio-data") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:59.552379 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.552354 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:59.553697 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.553669 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:59.553835 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.553742 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a87b9408-8c39-498b-906b-f664cea0d922-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:25:59.553835 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.553776 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-kube-api-access-knpb8" (OuterVolumeSpecName: "kube-api-access-knpb8") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "kube-api-access-knpb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:25:59.553968 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.553954 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-istio-token" (OuterVolumeSpecName: "istio-token") pod "a87b9408-8c39-498b-906b-f664cea0d922" (UID: "a87b9408-8c39-498b-906b-f664cea0d922"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:25:59.650007 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.649974 2574 generic.go:356] "Generic (PLEG): container finished" podID="a87b9408-8c39-498b-906b-f664cea0d922" containerID="729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153" exitCode=0 Jul 06 17:25:59.650190 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.650021 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" Jul 06 17:25:59.650190 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.650023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" event={"ID":"a87b9408-8c39-498b-906b-f664cea0d922","Type":"ContainerDied","Data":"729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153"} Jul 06 17:25:59.650190 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.650166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x" event={"ID":"a87b9408-8c39-498b-906b-f664cea0d922","Type":"ContainerDied","Data":"025218c2b7f6dc957dd5a7fc51fedaaaa07192c15561fec290a23756cfec0fbc"} Jul 06 17:25:59.650311 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.650199 2574 scope.go:117] "RemoveContainer" containerID="729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153" Jul 06 17:25:59.652630 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.652607 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-envoy\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.652741 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.652633 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-istio-token\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.652741 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.652647 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-workload-socket\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.652741 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.652661 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knpb8\" (UniqueName: \"kubernetes.io/projected/a87b9408-8c39-498b-906b-f664cea0d922-kube-api-access-knpb8\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.652741 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.652675 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a87b9408-8c39-498b-906b-f664cea0d922-istio-podinfo\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.652741 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.652688 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a87b9408-8c39-498b-906b-f664cea0d922-istiod-ca-cert\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.652741 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.652701 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a87b9408-8c39-498b-906b-f664cea0d922-istio-data\") on node \"ip-10-0-130-239.ec2.internal\" DevicePath \"\"" Jul 06 17:25:59.660016 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.659999 2574 scope.go:117] "RemoveContainer" containerID="0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215" Jul 06 17:25:59.668262 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.668237 2574 scope.go:117] "RemoveContainer" containerID="729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153" Jul 06 17:25:59.668546 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:25:59.668524 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153\": container with ID starting with 729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153 not found: ID does not exist" containerID="729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153" Jul 06 17:25:59.668623 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.668552 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153"} err="failed to get container status \"729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153\": rpc error: code = NotFound desc = could not find container \"729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153\": container with ID starting with 729241635a7862b3467ca397221f7a1ca649fbb7331d6ac2540724ca1fb2f153 not found: ID does not exist" Jul 06 17:25:59.668623 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.668568 2574 scope.go:117] "RemoveContainer" containerID="0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215" Jul 06 17:25:59.668823 ip-10-0-130-239 kubenswrapper[2574]: E0706 17:25:59.668809 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215\": container with ID starting with 0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215 not found: ID does not exist" containerID="0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215" Jul 06 17:25:59.668862 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.668826 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215"} err="failed to get container status \"0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215\": rpc error: code = NotFound desc = could not find container \"0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215\": container with ID starting with 0b00ba76db2ec2e4b45d51a5f21723c4a16d6bf270030ad071fa9b28da507215 not found: ID does not exist" Jul 06 17:25:59.676810 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.676783 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x"] Jul 06 17:25:59.678397 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.678378 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-0ebdd9-openshift-default-7b5bd674cf-qkf4x"] Jul 06 17:25:59.748260 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.748230 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87b9408-8c39-498b-906b-f664cea0d922" path="/var/lib/kubelet/pods/a87b9408-8c39-498b-906b-f664cea0d922/volumes" Jul 06 17:25:59.788632 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.788606 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-78ddc7f974-4rtm2_03c508e2-5ff7-46aa-9e63-c52efcb1128d/maas-api/0.log" Jul 06 17:25:59.909772 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:25:59.909747 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722620-fqnlp_539fd698-c3a4-4a12-80bd-87e52e2c7246/cleanup/0.log" Jul 06 17:26:00.019866 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:00.019796 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722635-md567_c7b17d09-4c54-478c-9c89-805c025d9230/cleanup/0.log" Jul 06 17:26:00.139304 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:00.139274 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-589c49595b-55qhp_61322c1f-7756-4293-a5c6-c2df5f79e4ed/manager/0.log" Jul 06 17:26:00.614441 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:00.614413 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d66bcd77d-ffxdz_c0ea03ea-d2c5-46db-86e7-48012d5a16b7/copy-manifests/0.log" Jul 06 17:26:00.620836 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:00.620812 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d66bcd77d-ffxdz_c0ea03ea-d2c5-46db-86e7-48012d5a16b7/manager/0.log" Jul 06 17:26:02.929184 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:02.929157 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-c9nxd_e764667a-8a18-4668-8e93-323e6383a810/manager/0.log" Jul 06 17:26:03.383158 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:03.383131 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hqzt9_74bd927a-4a82-47c4-b177-860aa330b4dd/discovery/0.log" Jul 06 17:26:03.865424 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:03.865377 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-8mnwv_71c04877-970a-411a-90de-5311266f5cc0/payload-pre-processing/0.log" Jul 06 17:26:04.793146 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:04.793121 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbv6ml_13864aa8-03dc-4572-96a2-199bffc5a203/main/0.log" Jul 06 17:26:09.129804 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:09.129757 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:26:09.197033 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:09.197001 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:26:12.494372 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:12.494341 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xrpmh_79a6f67d-6571-4bbe-92e9-d640eb94ac19/global-pull-secret-syncer/0.log" Jul 06 17:26:12.551500 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:12.551470 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" Jul 06 17:26:12.686001 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:12.685974 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-239.ec2.internal_1115f28ae51a4317d8862e47dd2c12eb/haproxy/0.log" Jul 06 17:26:18.559080 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:18.559053 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-c9nxd_e764667a-8a18-4668-8e93-323e6383a810/manager/0.log" Jul 06 17:26:20.345206 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.345175 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1f505df-8e0b-41ce-9564-dccd86f560a5/alertmanager/0.log" Jul 06 17:26:20.374922 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.374896 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1f505df-8e0b-41ce-9564-dccd86f560a5/config-reloader/0.log" Jul 06 17:26:20.396893 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.396824 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1f505df-8e0b-41ce-9564-dccd86f560a5/kube-rbac-proxy-web/0.log" Jul 06 17:26:20.428769 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.428743 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1f505df-8e0b-41ce-9564-dccd86f560a5/kube-rbac-proxy/0.log" Jul 06 17:26:20.453824 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.453800 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1f505df-8e0b-41ce-9564-dccd86f560a5/kube-rbac-proxy-metric/0.log" Jul 06 17:26:20.474519 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.474497 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1f505df-8e0b-41ce-9564-dccd86f560a5/prom-label-proxy/0.log" Jul 06 17:26:20.503227 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.503183 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1f505df-8e0b-41ce-9564-dccd86f560a5/init-config-reloader/0.log" Jul 06 17:26:20.651629 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.651558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66b9d4f887-5xftb_d7e15d77-2583-4602-b954-7ebee2216a26/metrics-server/0.log" Jul 06 17:26:20.801261 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.801233 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qxkpx_eb01898e-d0ac-49f4-ba1a-b33e9b4a2422/node-exporter/0.log" Jul 06 17:26:20.822710 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.822676 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qxkpx_eb01898e-d0ac-49f4-ba1a-b33e9b4a2422/kube-rbac-proxy/0.log" Jul 06 17:26:20.845173 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.845151 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qxkpx_eb01898e-d0ac-49f4-ba1a-b33e9b4a2422/init-textfile/0.log" Jul 06 17:26:20.955664 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.955590 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-vgx4v_eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a/kube-rbac-proxy-main/0.log" Jul 06 17:26:20.992993 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:20.992965 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-vgx4v_eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a/kube-rbac-proxy-self/0.log" Jul 06 17:26:21.025476 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.025419 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-vgx4v_eeec9a9b-b9f7-430a-a5d5-e23ffec7f44a/openshift-state-metrics/0.log" Jul 06 17:26:21.070035 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.070014 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eaeddba-2751-4fbd-8bde-6a3a90f6f07e/prometheus/0.log" Jul 06 17:26:21.096912 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.096889 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eaeddba-2751-4fbd-8bde-6a3a90f6f07e/config-reloader/0.log" Jul 06 17:26:21.124369 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.124341 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eaeddba-2751-4fbd-8bde-6a3a90f6f07e/thanos-sidecar/0.log" Jul 06 17:26:21.164263 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.164240 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eaeddba-2751-4fbd-8bde-6a3a90f6f07e/kube-rbac-proxy-web/0.log" Jul 06 17:26:21.234135 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.234108 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eaeddba-2751-4fbd-8bde-6a3a90f6f07e/kube-rbac-proxy/0.log" Jul 06 17:26:21.270706 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.270682 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eaeddba-2751-4fbd-8bde-6a3a90f6f07e/kube-rbac-proxy-thanos/0.log" Jul 06 17:26:21.320014 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.319992 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eaeddba-2751-4fbd-8bde-6a3a90f6f07e/init-config-reloader/0.log" Jul 06 17:26:21.530142 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.530067 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-546c965ccc-8nzkf_77755948-4df5-46cd-af5f-c99a51777479/telemeter-client/0.log" Jul 06 17:26:21.560651 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.560628 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-546c965ccc-8nzkf_77755948-4df5-46cd-af5f-c99a51777479/reload/0.log" Jul 06 17:26:21.594576 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:21.594547 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-546c965ccc-8nzkf_77755948-4df5-46cd-af5f-c99a51777479/kube-rbac-proxy/0.log" Jul 06 17:26:22.249133 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249093 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5"] Jul 06 17:26:22.249603 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249584 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249606 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249622 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249630 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249639 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249646 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249657 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="528f3502-4137-462c-87e3-1f7b373acab2" containerName="maas-api" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249666 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="528f3502-4137-462c-87e3-1f7b373acab2" containerName="maas-api" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249695 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50926d06-69d8-44a1-a937-6bd435d74726" containerName="maas-api" Jul 06 17:26:22.249719 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249703 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="50926d06-69d8-44a1-a937-6bd435d74726" containerName="maas-api" Jul 06 17:26:22.250157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249789 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe27f9c-0f47-4c29-ac62-6da6d02f94a4" containerName="istio-proxy" Jul 06 17:26:22.250157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249803 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="528f3502-4137-462c-87e3-1f7b373acab2" containerName="maas-api" Jul 06 17:26:22.250157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249816 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" Jul 06 17:26:22.250157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249825 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87b9408-8c39-498b-906b-f664cea0d922" containerName="istio-proxy" Jul 06 17:26:22.250157 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.249838 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="50926d06-69d8-44a1-a937-6bd435d74726" containerName="maas-api" Jul 06 17:26:22.253786 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.253762 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.256492 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.256440 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hztcg\"/\"default-dockercfg-7dvzm\"" Jul 06 17:26:22.256601 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.256440 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hztcg\"/\"kube-root-ca.crt\"" Jul 06 17:26:22.257301 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.257277 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hztcg\"/\"openshift-service-ca.crt\"" Jul 06 17:26:22.271448 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.271423 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5"] Jul 06 17:26:22.354874 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.354844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7kxf\" (UniqueName: \"kubernetes.io/projected/3996e91a-28ca-475a-8009-8ac35c27d54c-kube-api-access-s7kxf\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.355082 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.354882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-proc\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.355082 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.354906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-sys\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.355082 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.354929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-lib-modules\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.355082 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.354995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-podres\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456476 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7kxf\" (UniqueName: \"kubernetes.io/projected/3996e91a-28ca-475a-8009-8ac35c27d54c-kube-api-access-s7kxf\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456648 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-proc\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456648 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-sys\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456648 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-lib-modules\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456648 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-podres\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456648 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456617 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-proc\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456648 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456638 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-sys\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456867 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-lib-modules\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.456867 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.456711 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3996e91a-28ca-475a-8009-8ac35c27d54c-podres\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.468178 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.468152 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7kxf\" (UniqueName: \"kubernetes.io/projected/3996e91a-28ca-475a-8009-8ac35c27d54c-kube-api-access-s7kxf\") pod \"perf-node-gather-daemonset-pfpl5\" (UID: \"3996e91a-28ca-475a-8009-8ac35c27d54c\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.564490 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.564382 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:22.709267 ip-10-0-130-239 kubenswrapper[2574]: W0706 17:26:22.709223 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3996e91a_28ca_475a_8009_8ac35c27d54c.slice/crio-beb5a53fc805c58b72f4b09793d40bf6475f6bdb61116372df5c79f69242e284 WatchSource:0}: Error finding container beb5a53fc805c58b72f4b09793d40bf6475f6bdb61116372df5c79f69242e284: Status 404 returned error can't find the container with id beb5a53fc805c58b72f4b09793d40bf6475f6bdb61116372df5c79f69242e284 Jul 06 17:26:22.709438 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.709324 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5"] Jul 06 17:26:22.764434 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:22.764405 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" event={"ID":"3996e91a-28ca-475a-8009-8ac35c27d54c","Type":"ContainerStarted","Data":"beb5a53fc805c58b72f4b09793d40bf6475f6bdb61116372df5c79f69242e284"} Jul 06 17:26:23.770386 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:23.770354 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" event={"ID":"3996e91a-28ca-475a-8009-8ac35c27d54c","Type":"ContainerStarted","Data":"05973afe39b635edb4f75bf99e762bc7d23cd47702bf0970e79df873d0b12bc9"} Jul 06 17:26:23.770797 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:23.770495 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:23.819251 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:23.819197 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" podStartSLOduration=1.8191797840000001 podStartE2EDuration="1.819179784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:26:23.81767518 +0000 UTC m=+2880.587566715" watchObservedRunningTime="2026-07-06 17:26:23.819179784 +0000 UTC m=+2880.589071324" Jul 06 17:26:26.698643 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:26.698608 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-srdrb_130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6/dns/0.log" Jul 06 17:26:26.739513 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:26.739482 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-srdrb_130b3c20-a6bf-4ed4-a3ee-220fd30a3cc6/kube-rbac-proxy/0.log" Jul 06 17:26:26.770737 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:26.770706 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jpgx_9b13ab05-cd28-4ff2-9b5f-92d2c511bff5/dns-node-resolver/0.log" Jul 06 17:26:27.535023 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:27.534994 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-768bdd6975-2lgqz_00a1230c-44c3-49a0-85d7-e111c5852469/registry/0.log" Jul 06 17:26:27.637265 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:27.637237 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nflws_c75e82f6-540f-4d90-be0a-ed9c9445152f/node-ca/0.log" Jul 06 17:26:28.828237 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:28.828206 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hqzt9_74bd927a-4a82-47c4-b177-860aa330b4dd/discovery/0.log" Jul 06 17:26:28.995732 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:28.995705 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-8mnwv_71c04877-970a-411a-90de-5311266f5cc0/payload-pre-processing/0.log" Jul 06 17:26:29.784282 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:29.784250 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-pfpl5" Jul 06 17:26:29.812865 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:29.812839 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4nrv9_720b190b-59b6-42bb-846c-73437ca1229d/serve-healthcheck-canary/0.log" Jul 06 17:26:30.397863 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:30.397834 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dwrlf_86e55149-b176-417f-a475-35030db94939/kube-rbac-proxy/0.log" Jul 06 17:26:30.422233 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:30.422206 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dwrlf_86e55149-b176-417f-a475-35030db94939/exporter/0.log" Jul 06 17:26:30.462143 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:30.462112 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dwrlf_86e55149-b176-417f-a475-35030db94939/extractor/0.log" Jul 06 17:26:33.179869 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:33.179833 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hqzt9_74bd927a-4a82-47c4-b177-860aa330b4dd/discovery/0.log" Jul 06 17:26:33.595945 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:33.595917 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-8mnwv_71c04877-970a-411a-90de-5311266f5cc0/payload-pre-processing/0.log" Jul 06 17:26:33.600027 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:33.600000 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-78ddc7f974-4rtm2_03c508e2-5ff7-46aa-9e63-c52efcb1128d/maas-api/0.log" Jul 06 17:26:33.640811 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:33.640786 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722620-fqnlp_539fd698-c3a4-4a12-80bd-87e52e2c7246/cleanup/0.log" Jul 06 17:26:33.718690 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:33.718616 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722635-md567_c7b17d09-4c54-478c-9c89-805c025d9230/cleanup/0.log" Jul 06 17:26:33.834503 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:33.834475 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-589c49595b-55qhp_61322c1f-7756-4293-a5c6-c2df5f79e4ed/manager/0.log" Jul 06 17:26:34.240839 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:34.240801 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d66bcd77d-ffxdz_c0ea03ea-d2c5-46db-86e7-48012d5a16b7/manager/0.log" Jul 06 17:26:34.293483 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:34.293429 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d66bcd77d-ffxdz_c0ea03ea-d2c5-46db-86e7-48012d5a16b7/copy-manifests/0.log" Jul 06 17:26:35.056845 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:35.056814 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hqzt9_74bd927a-4a82-47c4-b177-860aa330b4dd/discovery/0.log" Jul 06 17:26:35.315383 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:35.315304 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-8mnwv_71c04877-970a-411a-90de-5311266f5cc0/payload-pre-processing/0.log" Jul 06 17:26:35.967751 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:35.967658 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c47b9fd86-hz9lb_cda095e1-cac1-4fdb-bad4-71643064b7e8/manager/0.log" Jul 06 17:26:37.465257 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:37.465227 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:26:37.528742 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:37.528708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5w5l8_3dcf852d-ac55-4a06-8597-f77c25cb65a5/konnectivity-agent/0.log" limit=1024 Jul 06 17:26:41.215389 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:41.215356 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-csndl_6be92524-de0c-4b62-9df8-28acd33362e7/migrator/0.log" Jul 06 17:26:41.238887 ip-10-0-130-239 kubenswrapper[2574]: I0706 17:26:41.238861 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-csndl_6be92524-de0c-4b62-9df8-28acd33362e7/graceful-termination/0.log"