Jul 29 19:00:17.770309 ip-10-0-142-16 systemd[1]: Starting Kubernetes Kubelet... Jul 29 19:00:18.171081 ip-10-0-142-16 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 19:00:18.171081 ip-10-0-142-16 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 19:00:18.171081 ip-10-0-142-16 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 19:00:18.171081 ip-10-0-142-16 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 19:00:18.171081 ip-10-0-142-16 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 19:00:18.172111 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.172024 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 19:00:18.177810 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177779 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 19:00:18.177922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177903 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 19:00:18.177922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177908 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 19:00:18.177922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177911 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 19:00:18.177922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177914 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 19:00:18.177922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177917 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 19:00:18.177922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177920 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177927 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177930 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177934 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177937 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177940 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177942 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177945 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177948 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177951 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177954 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177957 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177960 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177963 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177966 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177969 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177972 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177975 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177978 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177983 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 19:00:18.178092 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177986 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177989 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177992 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177995 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.177998 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178000 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178015 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178019 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178022 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178024 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178027 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178030 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178032 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178035 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178038 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178041 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178044 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178046 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178049 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 19:00:18.178573 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178054 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178059 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178062 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178066 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178069 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178073 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178076 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178079 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178082 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178085 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178088 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178091 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178093 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178096 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178099 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178102 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178105 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178107 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178110 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178113 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 19:00:18.179054 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178115 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178118 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178120 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178123 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178126 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178128 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178131 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178134 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178137 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178139 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178142 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178145 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178150 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178153 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178156 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178159 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178162 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178164 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178167 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 19:00:18.179580 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178170 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178173 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178175 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178178 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178181 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178184 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178187 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178190 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178194 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178196 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178199 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178202 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178205 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178208 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178210 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178981 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178989 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178992 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178995 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.178997 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 19:00:18.180050 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179000 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179017 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179020 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179023 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179026 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179028 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179031 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179033 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179036 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179039 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179041 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179044 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179048 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179052 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179054 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179057 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179060 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179063 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179066 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179069 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 19:00:18.180524 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179072 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179074 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179077 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179079 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179082 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179085 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179087 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179090 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179092 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179095 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179097 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179100 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179103 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179106 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179109 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179111 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179114 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179116 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179119 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179122 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 19:00:18.181039 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179124 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179127 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179129 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179132 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179134 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179137 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179140 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179143 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179146 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179151 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179154 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179157 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179160 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179162 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179165 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179167 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179171 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179174 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179176 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179179 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 19:00:18.181521 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179182 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179185 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179187 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179190 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179193 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179196 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179199 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179202 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179204 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179212 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179216 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179219 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179222 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179225 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179227 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179230 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179233 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179236 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179239 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 19:00:18.182017 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179242 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179245 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179247 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179250 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179252 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179255 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179258 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179262 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179265 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179269 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179272 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179275 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179278 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179280 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179283 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179682 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179687 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179690 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179692 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179695 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 19:00:18.182485 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179698 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179701 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179704 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179706 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179709 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179712 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179717 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179720 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179724 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179727 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179730 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179733 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179736 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179739 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179741 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179744 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179746 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179749 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179751 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 19:00:18.182977 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179754 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179756 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179758 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179761 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179763 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179766 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179768 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179771 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179773 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179776 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179779 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179782 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179784 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179787 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179790 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179794 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179797 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179800 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179803 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 19:00:18.183451 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179805 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179808 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179810 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179813 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179817 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179820 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179824 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179827 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179830 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179832 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179835 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179838 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179841 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179843 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179846 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179849 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179852 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179855 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179857 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179860 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 29 19:00:18.183922 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179862 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179865 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179867 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179871 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179873 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179876 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179878 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179881 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179883 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179887 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179890 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179893 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179895 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179898 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179900 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179903 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179905 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179907 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179910 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179913 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 19:00:18.184453 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179916 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179918 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179920 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179923 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179925 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179928 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179930 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179933 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179935 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179938 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179940 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179943 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179945 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179948 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179950 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.179953 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180035 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180042 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180049 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180055 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180059 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 19:00:18.184931 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180063 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180067 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180072 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180076 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180079 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180082 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180085 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180089 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180092 2575 flags.go:64] FLAG: --cgroup-root="" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180094 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180097 2575 flags.go:64] FLAG: --client-ca-file="" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180101 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180104 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180108 2575 flags.go:64] FLAG: --cluster-domain="" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180111 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180114 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180117 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180120 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180124 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180127 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180130 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180133 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180136 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180139 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 19:00:18.185446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180142 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180148 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180151 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180155 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180159 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180161 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180164 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180167 2575 flags.go:64] FLAG: --enable-server="true" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180170 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180175 2575 flags.go:64] FLAG: --event-burst="100" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180178 2575 flags.go:64] FLAG: --event-qps="50" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180181 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180184 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180187 2575 flags.go:64] FLAG: --eviction-hard="" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180191 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180194 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180197 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180199 2575 flags.go:64] FLAG: --eviction-soft="" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180202 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180205 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180212 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180215 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180218 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180220 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180223 2575 flags.go:64] FLAG: --feature-gates="" Jul 29 19:00:18.186033 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180227 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180230 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180233 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180236 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180239 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180242 2575 flags.go:64] FLAG: --help="false" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180245 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180248 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180252 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180255 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180258 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180261 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180264 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180267 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180270 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180272 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180275 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180278 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180281 2575 flags.go:64] FLAG: --kube-reserved="" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180284 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180287 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180290 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180293 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180297 2575 flags.go:64] FLAG: --lock-file="" Jul 29 19:00:18.186767 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180300 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180303 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180306 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180312 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180315 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180318 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180321 2575 flags.go:64] FLAG: --logging-format="text" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180324 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180327 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180329 2575 flags.go:64] FLAG: --manifest-url="" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180333 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180337 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180340 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180344 2575 flags.go:64] FLAG: --max-pods="110" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180347 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180349 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180353 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180356 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180359 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180362 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180365 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180377 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180380 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180383 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 19:00:18.187408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180386 2575 flags.go:64] FLAG: --pod-cidr="" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180389 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180394 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180397 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180400 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180403 2575 flags.go:64] FLAG: --port="10250" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180406 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180410 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c8d42c1a18670211" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180414 2575 flags.go:64] FLAG: --qos-reserved="" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180417 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180420 2575 flags.go:64] FLAG: --register-node="true" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180422 2575 flags.go:64] FLAG: --register-with-taints="" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180426 2575 flags.go:64] FLAG: --registry-burst="10" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180430 2575 flags.go:64] FLAG: --registry-qps="5" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180433 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180436 2575 flags.go:64] FLAG: --reserved-memory="" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180439 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180442 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180445 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180448 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180451 2575 flags.go:64] FLAG: --runonce="false" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180454 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180457 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180460 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180463 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 19:00:18.188043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180467 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180470 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180473 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180476 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180479 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180482 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180484 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180487 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180490 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180493 2575 flags.go:64] FLAG: --system-cgroups="" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180496 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180501 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180504 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180507 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180512 2575 flags.go:64] FLAG: --tls-min-version="" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180515 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180518 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180521 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180524 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180527 2575 flags.go:64] FLAG: --v="2" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180531 2575 flags.go:64] FLAG: --version="false" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180536 2575 flags.go:64] FLAG: --vmodule="" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180540 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180543 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180662 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 19:00:18.188644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180666 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180669 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180672 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180674 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180677 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180679 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180682 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180686 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180689 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180692 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180694 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180697 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180699 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180702 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180705 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180707 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180710 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180713 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180715 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180718 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 19:00:18.189271 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180721 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180724 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180727 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180730 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180732 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180735 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180737 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180740 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180742 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180746 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180748 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180751 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180753 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180756 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180760 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180763 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180766 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180769 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180772 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180776 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 19:00:18.189756 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180778 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180781 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180783 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180786 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180788 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180793 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180797 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180799 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180802 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180804 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180807 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180809 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180812 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180816 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180818 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180821 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180823 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180826 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180829 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180831 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 19:00:18.190266 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180834 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180837 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180840 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180842 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180845 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180847 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180850 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180852 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180855 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180857 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180861 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180866 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180869 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180871 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180874 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180877 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180879 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180881 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180884 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 19:00:18.190839 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180886 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180889 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180891 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180894 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180897 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180899 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180903 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180905 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180908 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180911 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180913 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180916 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180918 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180921 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180923 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180926 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180929 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180931 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 19:00:18.191330 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.180934 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.180939 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.187822 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.187839 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.188727 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.188735 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188795 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188801 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188806 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188809 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188813 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188816 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188819 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188822 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 19:00:18.191769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188824 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188827 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188830 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188833 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188835 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188838 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188841 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188843 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188846 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188848 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188851 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188854 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188856 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188859 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188861 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188864 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188867 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188869 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188872 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 19:00:18.192150 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188875 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188878 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188882 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188885 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188887 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188890 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188893 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188896 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188898 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188901 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188904 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188907 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188909 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188912 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188915 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188917 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188920 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188922 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188925 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 19:00:18.192623 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188928 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188930 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188933 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188936 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188939 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188941 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188944 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188947 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188950 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188953 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188955 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188958 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188961 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188963 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188966 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188971 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188975 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188978 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188981 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188983 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 19:00:18.193098 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188986 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188989 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188991 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188994 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188997 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.188999 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189002 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189017 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189021 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189023 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189026 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189028 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189031 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189035 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189039 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189042 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189046 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189048 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189051 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189054 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 19:00:18.193644 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189057 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189060 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189062 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189065 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189068 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189071 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189074 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189077 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189079 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189082 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189085 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189088 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189091 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.189096 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189210 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189214 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 19:00:18.194153 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189217 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189220 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189223 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189226 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189228 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189231 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189234 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189236 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189239 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189242 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189244 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189247 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189250 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189253 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189255 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189258 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189260 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189263 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189265 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189268 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 19:00:18.194546 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189270 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189274 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189277 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189280 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189283 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189285 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189289 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189292 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189294 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189297 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189299 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189302 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189304 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189307 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189310 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189312 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189315 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189319 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189322 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189325 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 19:00:18.195056 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189328 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189331 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189334 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189337 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189341 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189343 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189346 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189349 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189352 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189355 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189358 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189360 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189363 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189366 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189369 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189372 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189375 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189377 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189381 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189385 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 19:00:18.195542 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189388 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189390 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189393 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189395 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189398 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189401 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189403 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189405 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189408 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189411 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189413 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189416 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189418 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189420 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189423 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189425 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189428 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189431 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189434 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189436 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 19:00:18.196136 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189438 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189441 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189445 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189448 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189451 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189454 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189457 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189460 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189463 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189465 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189468 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189471 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189474 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189477 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189479 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189482 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 19:00:18.196657 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:18.189484 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 19:00:18.197120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.189489 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 19:00:18.197120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.189619 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 19:00:18.197120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.191666 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 19:00:18.197120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.193192 2575 server.go:1019] "Starting client certificate rotation" Jul 29 19:00:18.197120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.193298 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 19:00:18.197120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.193370 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 19:00:18.197120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.196153 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 19:00:18.200794 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.200770 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 19:00:18.212444 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.212420 2575 log.go:25] "Validated CRI v1 runtime API" Jul 29 19:00:18.214791 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.214775 2575 log.go:25] "Validated CRI v1 image API" Jul 29 19:00:18.216848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.216830 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 19:00:18.219352 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.219324 2575 fs.go:135] Filesystem UUIDs: map[21b6f8a7-1e98-4057-a3c5-78af5006ba8c:/dev/nvme0n1p3 5c012295-c2c4-47c1-80b0-8a0539898b2e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 29 19:00:18.219416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.219352 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 19:00:18.223318 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.223300 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 19:00:18.225262 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.225153 2575 manager.go:216] Machine: {Timestamp:2026-07-29 19:00:18.223726088 +0000 UTC m=+0.339255115 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3071187 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25888b2e960264ec7ca1fb6ef472ad SystemUUID:ec25888b-2e96-0264-ec7c-a1fb6ef472ad BootID:e1033d3a-d0ba-499f-bd16-460ba1b901f8 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:97:d4:96:0c:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:97:d4:96:0c:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:cd:0b:d9:fb:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 19:00:18.225262 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.225261 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 19:00:18.225397 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.225386 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 19:00:18.228523 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.228496 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 19:00:18.228661 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.228525 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 19:00:18.229192 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.229182 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 19:00:18.229230 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.229195 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 19:00:18.229230 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.229202 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 19:00:18.229915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.229905 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 19:00:18.230771 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.230761 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 19:00:18.232482 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.232471 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 29 19:00:18.232599 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.232589 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 19:00:18.232628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.232620 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 29 19:00:18.232661 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.232642 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 19:00:18.232661 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.232658 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 19:00:18.232743 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.232671 2575 kubelet.go:398] "Adding apiserver pod source" Jul 29 19:00:18.232743 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.232681 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 19:00:18.233156 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.233144 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 19:00:18.233192 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.233170 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 19:00:18.235937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.235919 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 19:00:18.237785 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.237771 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 19:00:18.237849 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.237801 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 19:00:18.239758 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239746 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239762 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239768 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239773 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239785 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239796 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239806 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239811 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239817 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 19:00:18.239820 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239823 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 19:00:18.240215 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239840 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 19:00:18.240215 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.239845 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 19:00:18.240679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.240664 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 19:00:18.240721 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.240682 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 19:00:18.244132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.244117 2575 server.go:1268] "Started kubelet" Jul 29 19:00:18.244351 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.244223 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 19:00:18.244454 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.244373 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 19:00:18.244454 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.244407 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 19:00:18.244974 ip-10-0-142-16 systemd[1]: Started Kubernetes Kubelet. Jul 29 19:00:18.249568 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.249544 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 19:00:18.249695 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.249548 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 19:00:18.249742 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.249696 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 19:00:18.249742 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.249718 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 19:00:18.251160 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.251138 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 19:00:18.251359 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.251343 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 29 19:00:18.255772 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.254783 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-16.ec2.internal.18c6d7d66c9b979b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-16.ec2.internal,UID:ip-10-0-142-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-16.ec2.internal,},FirstTimestamp:2026-07-29 19:00:18.244089755 +0000 UTC m=+0.359618779,LastTimestamp:2026-07-29 19:00:18.244089755 +0000 UTC m=+0.359618779,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-16.ec2.internal,}" Jul 29 19:00:18.258937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.258922 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 19:00:18.259042 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.258938 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 19:00:18.259042 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.259031 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:18.259852 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.259837 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 19:00:18.259913 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.259839 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 19:00:18.259913 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.259867 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 19:00:18.259997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.259962 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 29 19:00:18.259997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.259971 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 29 19:00:18.260157 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.260145 2575 factory.go:153] Registering CRI-O factory Jul 29 19:00:18.260216 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.260162 2575 factory.go:223] Registration of the crio container factory successfully Jul 29 19:00:18.260216 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.260210 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 19:00:18.260298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.260219 2575 factory.go:55] Registering systemd factory Jul 29 19:00:18.260298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.260227 2575 factory.go:223] Registration of the systemd container factory successfully Jul 29 19:00:18.260298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.260250 2575 factory.go:103] Registering Raw factory Jul 29 19:00:18.260298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.260264 2575 manager.go:1196] Started watching for new ooms in manager Jul 29 19:00:18.261193 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.261176 2575 manager.go:349] Starting recovery of all containers Jul 29 19:00:18.270229 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.270209 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s75rj" Jul 29 19:00:18.271198 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.271171 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 19:00:18.271568 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.271535 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 19:00:18.271568 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.271561 2575 manager.go:354] Recovery completed Jul 29 19:00:18.275535 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.275515 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s75rj" Jul 29 19:00:18.276520 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.276507 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 19:00:18.278497 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.278482 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory" Jul 29 19:00:18.278572 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.278510 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 19:00:18.278572 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.278520 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID" Jul 29 19:00:18.279179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.279164 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 19:00:18.279179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.279178 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 19:00:18.279286 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.279167 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 19:00:18.279286 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.279192 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 29 19:00:18.279834 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.279818 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory" Jul 29 19:00:18.279883 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.279849 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 19:00:18.279883 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.279861 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID" Jul 29 19:00:18.280688 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.280675 2575 policy_none.go:49] "None policy: Start" Jul 29 19:00:18.280752 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.280694 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 19:00:18.280752 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.280707 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 19:00:18.281721 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.281709 2575 policy_none.go:47] "Start" Jul 29 19:00:18.322610 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.322586 2575 manager.go:337] "Starting Device Plugin manager" Jul 29 19:00:18.322743 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.322626 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 19:00:18.322743 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.322638 2575 server.go:87] "Starting device plugin registration server" Jul 29 19:00:18.322937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.322923 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 19:00:18.323023 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.322938 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 19:00:18.323085 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.323072 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 19:00:18.323173 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.323141 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 19:00:18.323173 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.323149 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 19:00:18.323580 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.323551 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 19:00:18.323685 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.323599 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:18.423133 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.423058 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 19:00:18.424977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.424956 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory" Jul 29 19:00:18.425086 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.424995 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 19:00:18.425086 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.425028 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID" Jul 29 19:00:18.425086 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.425062 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.435222 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.435204 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.435291 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.435231 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-16.ec2.internal\": node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:18.442855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.442825 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 19:00:18.443969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.443939 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 19:00:18.444061 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.443990 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 19:00:18.444061 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.444032 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 19:00:18.444158 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.444074 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 19:00:18.446468 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.446453 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 19:00:18.457604 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.457587 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:18.544367 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.544333 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal"] Jul 29 19:00:18.544476 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.544418 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 19:00:18.545439 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.545416 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory" Jul 29 19:00:18.545541 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.545449 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 19:00:18.545541 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.545465 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID" Jul 29 19:00:18.546819 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.546807 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 19:00:18.546949 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.546933 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.547003 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.546965 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 19:00:18.547749 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.547734 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory" Jul 29 19:00:18.547818 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.547735 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory" Jul 29 19:00:18.547818 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.547777 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 19:00:18.547818 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.547788 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID" Jul 29 19:00:18.547818 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.547758 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 19:00:18.547998 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.547840 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID" Jul 29 19:00:18.548892 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.548879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.548938 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.548900 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 19:00:18.549735 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.549720 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory" Jul 29 19:00:18.549794 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.549749 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 19:00:18.549794 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.549764 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID" Jul 29 19:00:18.558657 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.558639 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:18.561907 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.561891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a18dea17ef0eaa6e55da3906653d9b64-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"a18dea17ef0eaa6e55da3906653d9b64\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.561978 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.561920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a18dea17ef0eaa6e55da3906653d9b64-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"a18dea17ef0eaa6e55da3906653d9b64\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.561978 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.561946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf51505803dd721bf45ed7e8f52c4dcf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-16.ec2.internal\" (UID: \"bf51505803dd721bf45ed7e8f52c4dcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.576768 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.576745 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-16.ec2.internal\" not found" node="ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.581178 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.581160 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-16.ec2.internal\" not found" node="ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.658923 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.658873 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:18.662177 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.662154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a18dea17ef0eaa6e55da3906653d9b64-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"a18dea17ef0eaa6e55da3906653d9b64\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.662283 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.662189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a18dea17ef0eaa6e55da3906653d9b64-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"a18dea17ef0eaa6e55da3906653d9b64\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.662283 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.662216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf51505803dd721bf45ed7e8f52c4dcf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-16.ec2.internal\" (UID: \"bf51505803dd721bf45ed7e8f52c4dcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.662283 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.662262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf51505803dd721bf45ed7e8f52c4dcf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-16.ec2.internal\" (UID: \"bf51505803dd721bf45ed7e8f52c4dcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.662283 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.662270 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a18dea17ef0eaa6e55da3906653d9b64-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"a18dea17ef0eaa6e55da3906653d9b64\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.662432 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.662262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a18dea17ef0eaa6e55da3906653d9b64-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"a18dea17ef0eaa6e55da3906653d9b64\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.759664 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.759585 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:18.860077 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.860036 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:18.879373 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.879349 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.885432 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:18.885407 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" Jul 29 19:00:18.960659 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:18.960624 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:19.061101 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:19.061040 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:20.260099 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:20.260069 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:20.291383 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.291364 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 19:00:20.291860 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.291538 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 19:00:20.360253 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:20.360226 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:20.365280 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.365260 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 19:00:20.378578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.378546 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 18:55:19 +0000 UTC" deadline="2028-05-13 18:11:05.62971097 +0000 UTC" Jul 29 19:00:20.378578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.378571 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15695h10m45.251142831s" Jul 29 19:00:20.449435 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.449376 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 19:00:20.460550 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:20.460529 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found" Jul 29 19:00:20.554398 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.554370 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 19:00:20.558272 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.558256 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" Jul 29 19:00:20.570495 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.570474 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 19:00:20.571537 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.571525 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" Jul 29 19:00:20.579416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.579396 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 19:00:20.812791 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:20.812753 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf51505803dd721bf45ed7e8f52c4dcf.slice/crio-d937ea02347fa7aa22f6aa5204b0b0479f5ca419473d4637deb5553901c50e00 WatchSource:0}: Error finding container d937ea02347fa7aa22f6aa5204b0b0479f5ca419473d4637deb5553901c50e00: Status 404 returned error can't find the container with id d937ea02347fa7aa22f6aa5204b0b0479f5ca419473d4637deb5553901c50e00 Jul 29 19:00:20.818825 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:20.818808 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:00:21.332185 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.332150 2575 apiserver.go:52] "Watching apiserver" Jul 29 19:00:21.339878 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.339859 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 19:00:21.340226 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.340204 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-jmb68","kube-system/konnectivity-agent-hj2k8","kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q","openshift-cluster-node-tuning-operator/tuned-z4m58","openshift-dns/node-resolver-4zjgp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal","openshift-multus/multus-additional-cni-plugins-4p7vd","openshift-multus/multus-cz8jd","kube-system/global-pull-secret-syncer-xgvm6","openshift-image-registry/node-ca-qxrp2","openshift-multus/network-metrics-daemon-ldxw2","openshift-network-diagnostics/network-check-target-nrdz9","openshift-network-operator/iptables-alerter-kxhbm"] Jul 29 19:00:21.342477 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.342461 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.342564 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.342543 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:21.343557 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.343540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:21.343613 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.343597 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:21.344553 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.344541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.345699 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.345677 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.346858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.346807 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 19:00:21.346858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.346811 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 19:00:21.347027 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.346836 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-94z5l\"" Jul 29 19:00:21.347027 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.346876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.347121 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.347112 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 19:00:21.348164 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.348147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.348459 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.348439 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 19:00:21.348586 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.348553 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dkkdc\"" Jul 29 19:00:21.348851 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.348836 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 19:00:21.348900 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.348884 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 19:00:21.349500 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.349484 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tv2tn\"" Jul 29 19:00:21.349592 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.349502 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 19:00:21.349592 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.349560 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 19:00:21.350235 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.350219 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.350589 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.350569 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vzf4x\"" Jul 29 19:00:21.350673 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.350636 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 19:00:21.350713 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.350674 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 19:00:21.351626 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.351607 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.352472 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.352457 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 19:00:21.352472 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.352462 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 19:00:21.352690 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.352560 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-66c5q\"" Jul 29 19:00:21.352753 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.352738 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 19:00:21.352810 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.352791 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:21.352810 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.352798 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 19:00:21.353718 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.353694 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mnmnv\"" Jul 29 19:00:21.353786 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.353744 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 19:00:21.353786 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.353776 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 19:00:21.354370 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.354354 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.355926 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.355906 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 19:00:21.356040 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.356015 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2n7lp\"" Jul 29 19:00:21.356137 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.356027 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 19:00:21.356973 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.356546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.356973 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.356909 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 19:00:21.357244 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.357222 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 19:00:21.357329 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.357315 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7kp4n\"" Jul 29 19:00:21.357590 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.357544 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 19:00:21.357678 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.357609 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 19:00:21.358438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.358400 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 19:00:21.358571 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.358555 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 19:00:21.358943 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.358922 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 19:00:21.359016 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.358955 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jmbcz\"" Jul 29 19:00:21.359016 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.358932 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 19:00:21.359201 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.359185 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:21.359656 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.359629 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:21.359751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.359734 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 19:00:21.359841 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.359797 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 19:00:21.368086 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-sys-fs\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.368162 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-kubelet\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.368162 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-system-cni-dir\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.368226 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vq48\" (UniqueName: \"kubernetes.io/projected/b8eb708d-e411-4fc8-bca7-9781b260c738-kube-api-access-6vq48\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.368226 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysctl-d\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.368290 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-run\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.368290 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-k8s-cni-cncf-io\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.368290 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.368377 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovnkube-script-lib\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.368377 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368332 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/517c963d-5b76-463e-8fa8-1c851fbbab1b-iptables-alerter-script\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.368377 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d1c8b71-e806-48a9-9037-5fe23acc92ec-host\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.368461 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysctl-conf\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.368461 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368434 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79e547f8-144d-4038-b31d-f8d5c715e276-etc-tuned\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.368461 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79e547f8-144d-4038-b31d-f8d5c715e276-tmp\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.368584 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-log-socket\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.368584 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/517c963d-5b76-463e-8fa8-1c851fbbab1b-host-slash\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.368584 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjks\" (UniqueName: \"kubernetes.io/projected/517c963d-5b76-463e-8fa8-1c851fbbab1b-kube-api-access-5mjks\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.368584 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d1c8b71-e806-48a9-9037-5fe23acc92ec-serviceca\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.368733 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gcq\" (UniqueName: \"kubernetes.io/projected/79e547f8-144d-4038-b31d-f8d5c715e276-kube-api-access-s4gcq\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.368733 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-cni-netd\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.368733 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-host\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.368733 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-kubelet\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.368733 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-var-lib-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.368733 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4kt\" (UniqueName: \"kubernetes.io/projected/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-kube-api-access-6d4kt\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-lib-modules\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7384f2c-3515-4683-bfc3-a751638bf375-multus-daemon-config\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368834 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-cni-binary-copy\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hdh\" (UniqueName: \"kubernetes.io/projected/1815188e-0527-44ec-8aa3-c889efb12990-kube-api-access-h6hdh\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-slash\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-systemd\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhz7\" (UniqueName: \"kubernetes.io/projected/7d1c8b71-e806-48a9-9037-5fe23acc92ec-kube-api-access-4jhz7\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-systemd\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-sys\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.368987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-cni-multus\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.369000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-hostroot\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-etc-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp9xr\" (UniqueName: \"kubernetes.io/projected/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-kube-api-access-hp9xr\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7384f2c-3515-4683-bfc3-a751638bf375-cni-binary-copy\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-cnibin\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3f27f0f-faaa-4939-8867-64954f08e3d1-agent-certs\") pod \"konnectivity-agent-hj2k8\" (UID: \"a3f27f0f-faaa-4939-8867-64954f08e3d1\") " pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-socket-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-kubernetes\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkxq\" (UniqueName: \"kubernetes.io/projected/b7384f2c-3515-4683-bfc3-a751638bf375-kube-api-access-mtkxq\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-ovn\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-node-log\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3f27f0f-faaa-4939-8867-64954f08e3d1-konnectivity-ca\") pod \"konnectivity-agent-hj2k8\" (UID: \"a3f27f0f-faaa-4939-8867-64954f08e3d1\") " pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:21.369489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-device-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysconfig\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-netns\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40076bf8-469e-4e66-a9ba-85e5a8e876b5-kubelet-config\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-registration-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-var-lib-kubelet\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-cni-bin\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovn-node-metrics-cert\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-conf-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-systemd-units\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-cni-bin\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pdn\" (UniqueName: \"kubernetes.io/projected/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-kube-api-access-j9pdn\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8eb708d-e411-4fc8-bca7-9781b260c738-tmp-dir\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovnkube-config\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-modprobe-d\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-system-cni-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-multus-certs\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-os-release\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.369981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40076bf8-469e-4e66-a9ba-85e5a8e876b5-dbus\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8eb708d-e411-4fc8-bca7-9781b260c738-hosts-file\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-cni-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-os-release\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-socket-dir-parent\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370147 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370171 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-env-overrides\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-cnibin\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370205 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-etc-kubernetes\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.370611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.370220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-run-netns\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.392149 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.392130 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7g229" Jul 29 19:00:21.399003 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.398978 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7g229" Jul 29 19:00:21.459629 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.459602 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 19:00:21.470894 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.470866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3f27f0f-faaa-4939-8867-64954f08e3d1-agent-certs\") pod \"konnectivity-agent-hj2k8\" (UID: \"a3f27f0f-faaa-4939-8867-64954f08e3d1\") " pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:21.471030 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.470899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-socket-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.471030 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.470924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-kubernetes\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.471030 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.470945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkxq\" (UniqueName: \"kubernetes.io/projected/b7384f2c-3515-4683-bfc3-a751638bf375-kube-api-access-mtkxq\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.471030 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.470966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.471030 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-kubernetes\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471066 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-socket-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-ovn\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-node-log\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3f27f0f-faaa-4939-8867-64954f08e3d1-konnectivity-ca\") pod \"konnectivity-agent-hj2k8\" (UID: \"a3f27f0f-faaa-4939-8867-64954f08e3d1\") " pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-device-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-ovn\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysconfig\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-netns\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-device-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysconfig\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471194 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-node-log\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471265 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40076bf8-469e-4e66-a9ba-85e5a8e876b5-kubelet-config\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40076bf8-469e-4e66-a9ba-85e5a8e876b5-kubelet-config\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.471306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-registration-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-netns\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-registration-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-var-lib-kubelet\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471729 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-var-lib-kubelet\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-cni-bin\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471762 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-cni-bin\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovn-node-metrics-cert\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-conf-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471866 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-conf-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-systemd-units\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-cni-bin\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pdn\" (UniqueName: \"kubernetes.io/projected/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-kube-api-access-j9pdn\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471982 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-systemd-units\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.471990 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8eb708d-e411-4fc8-bca7-9781b260c738-tmp-dir\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovnkube-config\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-modprobe-d\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-system-cni-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-multus-certs\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-os-release\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3f27f0f-faaa-4939-8867-64954f08e3d1-konnectivity-ca\") pod \"konnectivity-agent-hj2k8\" (UID: \"a3f27f0f-faaa-4939-8867-64954f08e3d1\") " pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-cni-bin\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472218 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8eb708d-e411-4fc8-bca7-9781b260c738-tmp-dir\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-modprobe-d\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40076bf8-469e-4e66-a9ba-85e5a8e876b5-dbus\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472231 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-system-cni-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-os-release\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472343 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-multus-certs\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472380 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8eb708d-e411-4fc8-bca7-9781b260c738-hosts-file\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.472796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-cni-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472475 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40076bf8-469e-4e66-a9ba-85e5a8e876b5-dbus\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.472509 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472533 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472557 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-cni-dir\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472557 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8eb708d-e411-4fc8-bca7-9781b260c738-hosts-file\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-os-release\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.472631 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret podName:40076bf8-469e-4e66-a9ba-85e5a8e876b5 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:21.9726114 +0000 UTC m=+2.989632474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret") pod "global-pull-secret-syncer-xgvm6" (UID: "40076bf8-469e-4e66-a9ba-85e5a8e876b5") : object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-socket-dir-parent\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-os-release\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-multus-socket-dir-parent\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472703 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472693 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-env-overrides\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-cnibin\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-etc-kubernetes\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.473657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-run-netns\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472790 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovnkube-config\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-sys-fs\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-kubelet\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472840 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-run-netns\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-cnibin\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-system-cni-dir\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-etc-kubernetes\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vq48\" (UniqueName: \"kubernetes.io/projected/b8eb708d-e411-4fc8-bca7-9781b260c738-kube-api-access-6vq48\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472901 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-sys-fs\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysctl-d\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472919 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-kubelet\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-run\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-k8s-cni-cncf-io\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-system-cni-dir\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.472987 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysctl-d\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.474449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473021 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-run\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-run-k8s-cni-cncf-io\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovnkube-script-lib\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-env-overrides\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/517c963d-5b76-463e-8fa8-1c851fbbab1b-iptables-alerter-script\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d1c8b71-e806-48a9-9037-5fe23acc92ec-host\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysctl-conf\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79e547f8-144d-4038-b31d-f8d5c715e276-etc-tuned\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473144 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d1c8b71-e806-48a9-9037-5fe23acc92ec-host\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79e547f8-144d-4038-b31d-f8d5c715e276-tmp\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-sysctl-conf\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-log-socket\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/517c963d-5b76-463e-8fa8-1c851fbbab1b-host-slash\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjks\" (UniqueName: \"kubernetes.io/projected/517c963d-5b76-463e-8fa8-1c851fbbab1b-kube-api-access-5mjks\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d1c8b71-e806-48a9-9037-5fe23acc92ec-serviceca\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gcq\" (UniqueName: \"kubernetes.io/projected/79e547f8-144d-4038-b31d-f8d5c715e276-kube-api-access-s4gcq\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-cni-netd\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-host\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-kubelet\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-var-lib-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4kt\" (UniqueName: \"kubernetes.io/projected/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-kube-api-access-6d4kt\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473572 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovnkube-script-lib\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-lib-modules\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-cni-netd\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7384f2c-3515-4683-bfc3-a751638bf375-multus-daemon-config\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-cni-binary-copy\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hdh\" (UniqueName: \"kubernetes.io/projected/1815188e-0527-44ec-8aa3-c889efb12990-kube-api-access-h6hdh\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473990 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-slash\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-systemd\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhz7\" (UniqueName: \"kubernetes.io/projected/7d1c8b71-e806-48a9-9037-5fe23acc92ec-kube-api-access-4jhz7\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-systemd\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-sys\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-cni-multus\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.475848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-hostroot\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-etc-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp9xr\" (UniqueName: \"kubernetes.io/projected/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-kube-api-access-hp9xr\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7384f2c-3515-4683-bfc3-a751638bf375-cni-binary-copy\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-cnibin\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-cnibin\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474533 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-host\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474582 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-kubelet\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-var-lib-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474696 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.473641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/517c963d-5b76-463e-8fa8-1c851fbbab1b-iptables-alerter-script\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.474992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-lib-modules\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475203 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-ovn-node-metrics-cert\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-sys\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.476679 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-run-systemd\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475559 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7384f2c-3515-4683-bfc3-a751638bf375-multus-daemon-config\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-host-var-lib-cni-multus\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7384f2c-3515-4683-bfc3-a751638bf375-hostroot\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475758 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-log-socket\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1815188e-0527-44ec-8aa3-c889efb12990-cni-binary-copy\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475798 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1815188e-0527-44ec-8aa3-c889efb12990-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/517c963d-5b76-463e-8fa8-1c851fbbab1b-host-slash\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79e547f8-144d-4038-b31d-f8d5c715e276-etc-systemd\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.476142 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.476211 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:21.976192921 +0000 UTC m=+2.993214010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.476340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d1c8b71-e806-48a9-9037-5fe23acc92ec-serviceca\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.476342 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7384f2c-3515-4683-bfc3-a751638bf375-cni-binary-copy\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.476423 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-etc-openvswitch\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.475294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3f27f0f-faaa-4939-8867-64954f08e3d1-agent-certs\") pod \"konnectivity-agent-hj2k8\" (UID: \"a3f27f0f-faaa-4939-8867-64954f08e3d1\") " pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.476538 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-host-slash\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.476560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79e547f8-144d-4038-b31d-f8d5c715e276-tmp\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.477507 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.476618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79e547f8-144d-4038-b31d-f8d5c715e276-etc-tuned\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.479023 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.479003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkxq\" (UniqueName: \"kubernetes.io/projected/b7384f2c-3515-4683-bfc3-a751638bf375-kube-api-access-mtkxq\") pod \"multus-cz8jd\" (UID: \"b7384f2c-3515-4683-bfc3-a751638bf375\") " pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.480744 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.480724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vq48\" (UniqueName: \"kubernetes.io/projected/b8eb708d-e411-4fc8-bca7-9781b260c738-kube-api-access-6vq48\") pod \"node-resolver-4zjgp\" (UID: \"b8eb708d-e411-4fc8-bca7-9781b260c738\") " pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.480923 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.480905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pdn\" (UniqueName: \"kubernetes.io/projected/73b5b2af-0b02-44a9-aa76-ed39e0346fb9-kube-api-access-j9pdn\") pod \"ovnkube-node-jmb68\" (UID: \"73b5b2af-0b02-44a9-aa76-ed39e0346fb9\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.482207 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.482191 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4kt\" (UniqueName: \"kubernetes.io/projected/8d5cbceb-2d84-42d4-ad02-7f58b984ccd4-kube-api-access-6d4kt\") pod \"aws-ebs-csi-driver-node-gr89q\" (UID: \"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.485964 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.485944 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 19:00:21.486070 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.485971 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 19:00:21.486070 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.485986 2575 projected.go:196] Error preparing data for projected volume kube-api-access-clchr for pod openshift-network-diagnostics/network-check-target-nrdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:21.486070 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.486045 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr podName:756f70ce-31b9-476f-a791-9365fdbe68ae nodeName:}" failed. No retries permitted until 2026-07-29 19:00:21.986032172 +0000 UTC m=+3.003053269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-clchr" (UniqueName: "kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr") pod "network-check-target-nrdz9" (UID: "756f70ce-31b9-476f-a791-9365fdbe68ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:21.487578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.487556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp9xr\" (UniqueName: \"kubernetes.io/projected/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-kube-api-access-hp9xr\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:21.487816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.487795 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjks\" (UniqueName: \"kubernetes.io/projected/517c963d-5b76-463e-8fa8-1c851fbbab1b-kube-api-access-5mjks\") pod \"iptables-alerter-kxhbm\" (UID: \"517c963d-5b76-463e-8fa8-1c851fbbab1b\") " pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.488438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.488423 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gcq\" (UniqueName: \"kubernetes.io/projected/79e547f8-144d-4038-b31d-f8d5c715e276-kube-api-access-s4gcq\") pod \"tuned-z4m58\" (UID: \"79e547f8-144d-4038-b31d-f8d5c715e276\") " pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.488652 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.488637 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hdh\" (UniqueName: \"kubernetes.io/projected/1815188e-0527-44ec-8aa3-c889efb12990-kube-api-access-h6hdh\") pod \"multus-additional-cni-plugins-4p7vd\" (UID: \"1815188e-0527-44ec-8aa3-c889efb12990\") " pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.488788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.488772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhz7\" (UniqueName: \"kubernetes.io/projected/7d1c8b71-e806-48a9-9037-5fe23acc92ec-kube-api-access-4jhz7\") pod \"node-ca-qxrp2\" (UID: \"7d1c8b71-e806-48a9-9037-5fe23acc92ec\") " pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.547317 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.547265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" event={"ID":"bf51505803dd721bf45ed7e8f52c4dcf","Type":"ContainerStarted","Data":"d937ea02347fa7aa22f6aa5204b0b0479f5ca419473d4637deb5553901c50e00"} Jul 29 19:00:21.657257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.657154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kxhbm" Jul 29 19:00:21.664227 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.664205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4zjgp" Jul 29 19:00:21.669278 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:21.669080 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517c963d_5b76_463e_8fa8_1c851fbbab1b.slice/crio-db5b9f33c7124b29d4415c33e2b00cbe707f2042ecf071bb9c2f27bad0b77ca1 WatchSource:0}: Error finding container db5b9f33c7124b29d4415c33e2b00cbe707f2042ecf071bb9c2f27bad0b77ca1: Status 404 returned error can't find the container with id db5b9f33c7124b29d4415c33e2b00cbe707f2042ecf071bb9c2f27bad0b77ca1 Jul 29 19:00:21.671117 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.671093 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" Jul 29 19:00:21.674562 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:21.674209 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8eb708d_e411_4fc8_bca7_9781b260c738.slice/crio-e6afc38861110731be9bcd46b16377ed8d7c4ca3d34aca12ea647ed5d1e2a3e6 WatchSource:0}: Error finding container e6afc38861110731be9bcd46b16377ed8d7c4ca3d34aca12ea647ed5d1e2a3e6: Status 404 returned error can't find the container with id e6afc38861110731be9bcd46b16377ed8d7c4ca3d34aca12ea647ed5d1e2a3e6 Jul 29 19:00:21.675462 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.675297 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z4m58" Jul 29 19:00:21.681690 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.681664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cz8jd" Jul 29 19:00:21.686034 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.685796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" Jul 29 19:00:21.691190 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.691175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:21.695146 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:21.695086 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1815188e_0527_44ec_8aa3_c889efb12990.slice/crio-16026981d9a5f39a5623706ae43fed081ae3587d37d26289793de42ca3270658 WatchSource:0}: Error finding container 16026981d9a5f39a5623706ae43fed081ae3587d37d26289793de42ca3270658: Status 404 returned error can't find the container with id 16026981d9a5f39a5623706ae43fed081ae3587d37d26289793de42ca3270658 Jul 29 19:00:21.696364 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.696346 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:21.700095 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.699901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxrp2" Jul 29 19:00:21.700095 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:21.700031 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f27f0f_faaa_4939_8867_64954f08e3d1.slice/crio-eca36b008ac8878f6811d77c28fe36a679d9a6823d53914d9bee56a8dd7711ec WatchSource:0}: Error finding container eca36b008ac8878f6811d77c28fe36a679d9a6823d53914d9bee56a8dd7711ec: Status 404 returned error can't find the container with id eca36b008ac8878f6811d77c28fe36a679d9a6823d53914d9bee56a8dd7711ec Jul 29 19:00:21.706929 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:21.706903 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b5b2af_0b02_44a9_aa76_ed39e0346fb9.slice/crio-5ef820657c6f263b6b6a1c05bb4eb2901ba67012ca22551d43bcca4306706d27 WatchSource:0}: Error finding container 5ef820657c6f263b6b6a1c05bb4eb2901ba67012ca22551d43bcca4306706d27: Status 404 returned error can't find the container with id 5ef820657c6f263b6b6a1c05bb4eb2901ba67012ca22551d43bcca4306706d27 Jul 29 19:00:21.716038 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:21.716009 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1c8b71_e806_48a9_9037_5fe23acc92ec.slice/crio-79864b3e6961f1f8db1535673bc96a1e2ec9ac43ad463eefbca242bc98dccf77 WatchSource:0}: Error finding container 79864b3e6961f1f8db1535673bc96a1e2ec9ac43ad463eefbca242bc98dccf77: Status 404 returned error can't find the container with id 79864b3e6961f1f8db1535673bc96a1e2ec9ac43ad463eefbca242bc98dccf77 Jul 29 19:00:21.796200 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.796168 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 19:00:21.901510 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:21.901477 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18dea17ef0eaa6e55da3906653d9b64.slice/crio-cc55a02ce3a8c4119aae5150734ea694e346e6eecc46b64037d76b8578311eb0 WatchSource:0}: Error finding container cc55a02ce3a8c4119aae5150734ea694e346e6eecc46b64037d76b8578311eb0: Status 404 returned error can't find the container with id cc55a02ce3a8c4119aae5150734ea694e346e6eecc46b64037d76b8578311eb0 Jul 29 19:00:21.978637 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.978568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:21.978637 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:21.978624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:21.978789 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.978715 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:21.978789 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.978717 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:21.978789 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.978771 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret podName:40076bf8-469e-4e66-a9ba-85e5a8e876b5 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:22.978753769 +0000 UTC m=+3.995774855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret") pod "global-pull-secret-syncer-xgvm6" (UID: "40076bf8-469e-4e66-a9ba-85e5a8e876b5") : object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:21.978789 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:21.978785 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:22.978778461 +0000 UTC m=+3.995799535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:22.079847 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.079815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:22.079992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.079961 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 19:00:22.079992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.079977 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 19:00:22.079992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.079986 2575 projected.go:196] Error preparing data for projected volume kube-api-access-clchr for pod openshift-network-diagnostics/network-check-target-nrdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:22.080128 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.080037 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr podName:756f70ce-31b9-476f-a791-9365fdbe68ae nodeName:}" failed. No retries permitted until 2026-07-29 19:00:23.080021914 +0000 UTC m=+4.097042989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-clchr" (UniqueName: "kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr") pod "network-check-target-nrdz9" (UID: "756f70ce-31b9-476f-a791-9365fdbe68ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:22.401129 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.401037 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:55:21 +0000 UTC" deadline="2028-01-06 12:02:48.109328028 +0000 UTC" Jul 29 19:00:22.401129 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.401080 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12617h2m25.708251414s" Jul 29 19:00:22.543877 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.543766 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:22.544040 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.543891 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:22.550565 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.550532 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z4m58" event={"ID":"79e547f8-144d-4038-b31d-f8d5c715e276","Type":"ContainerStarted","Data":"2d367a8a0e0b97794a0adbe255b565d0cdaaf84784130aed4ef780d949113ee1"} Jul 29 19:00:22.552118 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.552084 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" event={"ID":"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4","Type":"ContainerStarted","Data":"46f69b94ae0503f8072ce3e4d48bb50ffea1cf3050b61bef55531a2318ce11c3"} Jul 29 19:00:22.553316 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.553292 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4zjgp" event={"ID":"b8eb708d-e411-4fc8-bca7-9781b260c738","Type":"ContainerStarted","Data":"e6afc38861110731be9bcd46b16377ed8d7c4ca3d34aca12ea647ed5d1e2a3e6"} Jul 29 19:00:22.554570 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.554545 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" event={"ID":"a18dea17ef0eaa6e55da3906653d9b64","Type":"ContainerStarted","Data":"cc55a02ce3a8c4119aae5150734ea694e346e6eecc46b64037d76b8578311eb0"} Jul 29 19:00:22.555932 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.555906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" event={"ID":"1815188e-0527-44ec-8aa3-c889efb12990","Type":"ContainerStarted","Data":"16026981d9a5f39a5623706ae43fed081ae3587d37d26289793de42ca3270658"} Jul 29 19:00:22.557260 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.557233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kxhbm" event={"ID":"517c963d-5b76-463e-8fa8-1c851fbbab1b","Type":"ContainerStarted","Data":"db5b9f33c7124b29d4415c33e2b00cbe707f2042ecf071bb9c2f27bad0b77ca1"} Jul 29 19:00:22.558605 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.558580 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxrp2" event={"ID":"7d1c8b71-e806-48a9-9037-5fe23acc92ec","Type":"ContainerStarted","Data":"79864b3e6961f1f8db1535673bc96a1e2ec9ac43ad463eefbca242bc98dccf77"} Jul 29 19:00:22.559738 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.559714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"5ef820657c6f263b6b6a1c05bb4eb2901ba67012ca22551d43bcca4306706d27"} Jul 29 19:00:22.561013 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.560990 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hj2k8" event={"ID":"a3f27f0f-faaa-4939-8867-64954f08e3d1","Type":"ContainerStarted","Data":"eca36b008ac8878f6811d77c28fe36a679d9a6823d53914d9bee56a8dd7711ec"} Jul 29 19:00:22.562286 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.562243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cz8jd" event={"ID":"b7384f2c-3515-4683-bfc3-a751638bf375","Type":"ContainerStarted","Data":"2a64f9dc2ae5f1ad84da339aa3b7bb303a2c453d9002e6e624089c9049684fba"} Jul 29 19:00:22.985825 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.985787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:22.985998 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:22.985869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:22.985998 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.985989 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:22.986118 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.986050 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret podName:40076bf8-469e-4e66-a9ba-85e5a8e876b5 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:24.986032723 +0000 UTC m=+6.003053800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret") pod "global-pull-secret-syncer-xgvm6" (UID: "40076bf8-469e-4e66-a9ba-85e5a8e876b5") : object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:22.986454 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.986433 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:22.986552 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:22.986488 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:24.986473619 +0000 UTC m=+6.003494698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:23.087262 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:23.087177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:23.087418 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:23.087385 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 19:00:23.087418 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:23.087404 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 19:00:23.087418 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:23.087416 2575 projected.go:196] Error preparing data for projected volume kube-api-access-clchr for pod openshift-network-diagnostics/network-check-target-nrdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:23.087598 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:23.087472 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr podName:756f70ce-31b9-476f-a791-9365fdbe68ae nodeName:}" failed. No retries permitted until 2026-07-29 19:00:25.087454284 +0000 UTC m=+6.104475371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-clchr" (UniqueName: "kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr") pod "network-check-target-nrdz9" (UID: "756f70ce-31b9-476f-a791-9365fdbe68ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:23.401739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:23.401636 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:55:21 +0000 UTC" deadline="2028-04-25 12:45:40.165371586 +0000 UTC" Jul 29 19:00:23.401739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:23.401681 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15257h45m16.763695083s" Jul 29 19:00:23.545177 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:23.545150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:23.545403 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:23.545282 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:23.545678 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:23.545659 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:23.545761 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:23.545748 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:23.583827 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:23.583726 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" event={"ID":"bf51505803dd721bf45ed7e8f52c4dcf","Type":"ContainerStarted","Data":"da0c249a06091411a14f09f7a6d0666cc77dffd6d44cc1172ecf657f9366ec00"} Jul 29 19:00:23.604074 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:23.604014 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" podStartSLOduration=3.603993449 podStartE2EDuration="3.603993449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:23.603495953 +0000 UTC m=+4.620517050" watchObservedRunningTime="2026-07-29 19:00:23.603993449 +0000 UTC m=+4.621014547" Jul 29 19:00:24.543980 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:24.543604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:24.544358 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:24.544027 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:25.004910 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:25.004824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:25.004910 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:25.004902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:25.005136 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.005048 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:25.005136 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.005111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:29.005091366 +0000 UTC m=+10.022112455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:25.005254 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.005173 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:25.005254 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.005213 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret podName:40076bf8-469e-4e66-a9ba-85e5a8e876b5 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:29.005201193 +0000 UTC m=+10.022222267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret") pod "global-pull-secret-syncer-xgvm6" (UID: "40076bf8-469e-4e66-a9ba-85e5a8e876b5") : object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:25.105875 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:25.105840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:25.106057 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.106015 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 19:00:25.106057 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.106039 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 19:00:25.106057 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.106052 2575 projected.go:196] Error preparing data for projected volume kube-api-access-clchr for pod openshift-network-diagnostics/network-check-target-nrdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:25.106211 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.106117 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr podName:756f70ce-31b9-476f-a791-9365fdbe68ae nodeName:}" failed. No retries permitted until 2026-07-29 19:00:29.106095693 +0000 UTC m=+10.123116770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-clchr" (UniqueName: "kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr") pod "network-check-target-nrdz9" (UID: "756f70ce-31b9-476f-a791-9365fdbe68ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:25.545684 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:25.545624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:25.546124 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.545758 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:25.546124 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:25.545624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:25.546232 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:25.546190 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:26.543856 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:26.543803 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:26.544029 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:26.543936 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:27.543233 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:27.543163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:27.543694 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:27.543317 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:27.543694 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:27.543370 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:27.543694 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:27.543435 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:28.542907 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:28.542832 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:28.543050 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:28.542978 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:29.039598 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:29.039568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:29.039949 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:29.039645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:29.039949 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.039774 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:29.039949 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.039840 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret podName:40076bf8-469e-4e66-a9ba-85e5a8e876b5 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:37.039820113 +0000 UTC m=+18.056841192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret") pod "global-pull-secret-syncer-xgvm6" (UID: "40076bf8-469e-4e66-a9ba-85e5a8e876b5") : object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:29.040241 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.040214 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:29.040313 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.040271 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:37.040255725 +0000 UTC m=+18.057276814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:29.140284 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:29.140245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:29.140444 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.140426 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 19:00:29.140498 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.140449 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 19:00:29.140498 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.140460 2575 projected.go:196] Error preparing data for projected volume kube-api-access-clchr for pod openshift-network-diagnostics/network-check-target-nrdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:29.140632 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.140530 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr podName:756f70ce-31b9-476f-a791-9365fdbe68ae nodeName:}" failed. No retries permitted until 2026-07-29 19:00:37.140497853 +0000 UTC m=+18.157518944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-clchr" (UniqueName: "kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr") pod "network-check-target-nrdz9" (UID: "756f70ce-31b9-476f-a791-9365fdbe68ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:29.546369 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:29.545622 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:29.546369 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.545737 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:29.546639 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:29.546486 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:29.546639 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:29.546623 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:30.543352 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:30.543314 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:30.543798 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:30.543466 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:31.546136 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:31.546103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:31.546617 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:31.546232 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:31.546617 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:31.546279 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:31.546617 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:31.546376 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:32.543720 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:32.543679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:32.543919 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:32.543819 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:33.546286 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:33.546257 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:33.546709 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:33.546296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:33.546709 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:33.546385 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:33.546709 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:33.546613 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:34.542799 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:34.542760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:34.542963 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:34.542897 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:35.543199 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:35.543158 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:35.543633 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:35.543204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:35.543633 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:35.543323 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:35.543633 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:35.543469 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:36.543722 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:36.543680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:36.544233 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:36.543805 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:37.105287 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:37.105251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:37.105449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:37.105326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:37.105449 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.105423 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:37.105578 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.105491 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret podName:40076bf8-469e-4e66-a9ba-85e5a8e876b5 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:53.10547394 +0000 UTC m=+34.122495020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret") pod "global-pull-secret-syncer-xgvm6" (UID: "40076bf8-469e-4e66-a9ba-85e5a8e876b5") : object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:37.105578 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.105425 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:37.105578 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.105550 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:53.10553967 +0000 UTC m=+34.122560760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:37.206164 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:37.206116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:37.206353 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.206264 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 19:00:37.206353 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.206287 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 19:00:37.206353 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.206297 2575 projected.go:196] Error preparing data for projected volume kube-api-access-clchr for pod openshift-network-diagnostics/network-check-target-nrdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:37.206353 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.206347 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr podName:756f70ce-31b9-476f-a791-9365fdbe68ae nodeName:}" failed. No retries permitted until 2026-07-29 19:00:53.206333703 +0000 UTC m=+34.223354776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-clchr" (UniqueName: "kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr") pod "network-check-target-nrdz9" (UID: "756f70ce-31b9-476f-a791-9365fdbe68ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:37.545921 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:37.545893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:37.546336 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:37.545893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:37.546336 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.546007 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:37.546336 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:37.546128 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:38.542852 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:38.542817 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:38.543022 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:38.542939 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:39.547383 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:39.547299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:39.547851 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:39.547591 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:39.548016 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:39.547992 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:39.548213 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:39.548166 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:40.543789 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:40.543750 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:40.543991 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:40.543872 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:41.544678 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.544472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:41.545410 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.544567 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:41.545410 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:41.544786 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:41.545410 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:41.544865 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:41.627409 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.627381 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxrp2" event={"ID":"7d1c8b71-e806-48a9-9037-5fe23acc92ec","Type":"ContainerStarted","Data":"ac877e349cc574efdbc81057eb778f0e0422c947cc092fa174b97eeb5edb3fdf"} Jul 29 19:00:41.630221 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.630194 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"a2c1d41e1c127fd38af685c6ee4a6ad66266c7ead10f8a7dc3acbdd858e8b4b2"} Jul 29 19:00:41.630314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.630279 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"654988a24382e65a7ab1591b9fc927c5ffc6c77816602758c41952bc66cfd9fb"} Jul 29 19:00:41.630364 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.630314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"bc8c74debcd9ae7e81d1de92339da53ce15d1886a5f1ec0cbd22a42b1e5f3a9d"} Jul 29 19:00:41.630364 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.630359 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"652a5e435ffe7aa9fb5e1bd4d0dda8f6e6d1198ffce8623e884d163e7ad4d31c"} Jul 29 19:00:41.631603 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.631581 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hj2k8" event={"ID":"a3f27f0f-faaa-4939-8867-64954f08e3d1","Type":"ContainerStarted","Data":"7d62965bb05aa533b91d9f3ccbac51149444e7354dbd1dc2fd681927b1ca4b67"} Jul 29 19:00:41.633038 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.633019 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cz8jd" event={"ID":"b7384f2c-3515-4683-bfc3-a751638bf375","Type":"ContainerStarted","Data":"e3cf83d7a78702c0132d2b7414059b088eca7796f9260bdea9a7849352fc8fea"} Jul 29 19:00:41.635345 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.635326 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z4m58" event={"ID":"79e547f8-144d-4038-b31d-f8d5c715e276","Type":"ContainerStarted","Data":"25239bd4ee9a7edb24a81d15e3f6426217430c8b76c29145a73ee373c5f570f2"} Jul 29 19:00:41.636802 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.636783 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" event={"ID":"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4","Type":"ContainerStarted","Data":"7e0aea26596de2b774a8fb935abe5d8137b6a3aa716dcc716254ed4dda3eb2f8"} Jul 29 19:00:41.638117 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.638097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4zjgp" event={"ID":"b8eb708d-e411-4fc8-bca7-9781b260c738","Type":"ContainerStarted","Data":"a35c38f4b677320d6ac04229c5e6637175d845cf18206c1825de913d9f146778"} Jul 29 19:00:41.639502 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.639483 2575 generic.go:356] "Generic (PLEG): container finished" podID="a18dea17ef0eaa6e55da3906653d9b64" containerID="bde655e0904299f0ee348740e97363fe28d16021715453066f2ee90d70e10498" exitCode=0 Jul 29 19:00:41.639599 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.639542 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" event={"ID":"a18dea17ef0eaa6e55da3906653d9b64","Type":"ContainerDied","Data":"bde655e0904299f0ee348740e97363fe28d16021715453066f2ee90d70e10498"} Jul 29 19:00:41.641064 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.641043 2575 generic.go:356] "Generic (PLEG): container finished" podID="1815188e-0527-44ec-8aa3-c889efb12990" containerID="744a2861938481a458bd934b2328d3dd2610587350b833187be6b02e7f9901f6" exitCode=0 Jul 29 19:00:41.641117 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.641078 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" event={"ID":"1815188e-0527-44ec-8aa3-c889efb12990","Type":"ContainerDied","Data":"744a2861938481a458bd934b2328d3dd2610587350b833187be6b02e7f9901f6"} Jul 29 19:00:41.656790 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.656751 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qxrp2" podStartSLOduration=3.376397896 podStartE2EDuration="22.65673814s" totalImagesPullingTime="19.280340244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:41.64106451 +0000 UTC m=+22.658085606" watchObservedRunningTime="2026-07-29 19:00:41.65673814 +0000 UTC m=+22.673759229" Jul 29 19:00:41.685793 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.685752 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z4m58" podStartSLOduration=3.368157304 podStartE2EDuration="22.685737705s" totalImagesPullingTime="19.317580401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:41.671141124 +0000 UTC m=+22.688162219" watchObservedRunningTime="2026-07-29 19:00:41.685737705 +0000 UTC m=+22.702758818" Jul 29 19:00:41.685899 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.685876 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4zjgp" podStartSLOduration=3.356707665 podStartE2EDuration="22.685866624s" totalImagesPullingTime="19.329158959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:41.685676998 +0000 UTC m=+22.702698095" watchObservedRunningTime="2026-07-29 19:00:41.685866624 +0000 UTC m=+22.702887722" Jul 29 19:00:41.709372 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.709336 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hj2k8" podStartSLOduration=8.600398349 podStartE2EDuration="22.709323316s" totalImagesPullingTime="14.108924967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:41.708879905 +0000 UTC m=+22.725901001" watchObservedRunningTime="2026-07-29 19:00:41.709323316 +0000 UTC m=+22.726344442" Jul 29 19:00:41.755255 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:41.755195 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cz8jd" podStartSLOduration=3.385488794 podStartE2EDuration="22.755175039s" totalImagesPullingTime="19.369686245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:41.731249667 +0000 UTC m=+22.748270776" watchObservedRunningTime="2026-07-29 19:00:41.755175039 +0000 UTC m=+22.772196136" Jul 29 19:00:42.542892 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.542860 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:42.543064 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:42.542994 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:42.576884 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.576841 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:42.577503 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.577486 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:42.645169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.645135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" event={"ID":"a18dea17ef0eaa6e55da3906653d9b64","Type":"ContainerStarted","Data":"eecc5a466ed603db3d83334a0ec2357d893bd745634b4c0ecacb5c79ca62117e"} Jul 29 19:00:42.647151 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.647123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kxhbm" event={"ID":"517c963d-5b76-463e-8fa8-1c851fbbab1b","Type":"ContainerStarted","Data":"2b9d3116f0c7989ab2e196b70e891c82bab159235875dac612bb3f3aa8f9c569"} Jul 29 19:00:42.650482 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.650452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"3a6ba66b1ac34846c1773233c8bed023ba24165cc045e42562e9569f923022f6"} Jul 29 19:00:42.650596 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.650540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"cdc447d929de405bd874b309d64a36fce87c41424ab9b472b486cc59abdbec2d"} Jul 29 19:00:42.651355 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.651158 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:42.651355 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.651316 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hj2k8" Jul 29 19:00:42.671164 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.671127 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" podStartSLOduration=22.671112891 podStartE2EDuration="22.671112891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:42.658469889 +0000 UTC m=+23.675490986" watchObservedRunningTime="2026-07-29 19:00:42.671112891 +0000 UTC m=+23.688133987" Jul 29 19:00:42.671528 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.671482 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kxhbm" podStartSLOduration=4.33441794 podStartE2EDuration="23.671471131s" totalImagesPullingTime="19.337053191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:42.670752228 +0000 UTC m=+23.687773323" watchObservedRunningTime="2026-07-29 19:00:42.671471131 +0000 UTC m=+23.688492228" Jul 29 19:00:42.811086 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:42.811064 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 19:00:43.439620 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:43.439501 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T19:00:42.811079898Z","UUID":"a040e771-55a9-4b4b-8a22-de769da643db","Handler":null,"Name":"","Endpoint":""} Jul 29 19:00:43.441053 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:43.441024 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 19:00:43.441183 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:43.441072 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 19:00:43.543436 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:43.543404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:43.543648 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:43.543563 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:43.543648 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:43.543589 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:43.543764 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:43.543691 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:43.654710 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:43.654672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" event={"ID":"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4","Type":"ContainerStarted","Data":"ffe605db4e5471b9e54eb1a9438ab4e27fb03532d0c32331f1f45b202a79dbff"} Jul 29 19:00:44.543366 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:44.543326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:44.543583 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:44.543445 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:44.661132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:44.661090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"77e3f6cca881eb2db0659b1bb53fa8d68aefff6e733e071813e7e6e6edf66083"} Jul 29 19:00:45.542949 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:45.542911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:45.543126 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:45.542958 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:45.543126 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:45.543098 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:45.543205 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:45.543162 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:46.543090 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.542883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:46.543448 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:46.543204 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:46.668968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.668880 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" event={"ID":"73b5b2af-0b02-44a9-aa76-ed39e0346fb9","Type":"ContainerStarted","Data":"e12f9cdf388bfb968b37967bddca58b2be5df4dc7aa8c26d827faa179ecb86f6"} Jul 29 19:00:46.669157 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.669139 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:46.669219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.669213 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:46.671044 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.671017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" event={"ID":"8d5cbceb-2d84-42d4-ad02-7f58b984ccd4","Type":"ContainerStarted","Data":"0317a6dfa0b57c2341dd189da1d67e84ace94684c19371b4b273db5c02218c11"} Jul 29 19:00:46.672796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.672769 2575 generic.go:356] "Generic (PLEG): container finished" podID="1815188e-0527-44ec-8aa3-c889efb12990" containerID="2af520b8d65773edc3072812a05f3f03ca37c24680a4910b96fc46fb9f1ef2ab" exitCode=0 Jul 29 19:00:46.672879 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.672810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" event={"ID":"1815188e-0527-44ec-8aa3-c889efb12990","Type":"ContainerDied","Data":"2af520b8d65773edc3072812a05f3f03ca37c24680a4910b96fc46fb9f1ef2ab"} Jul 29 19:00:46.684489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.684470 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:46.700662 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.700620 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" podStartSLOduration=8.304754972 podStartE2EDuration="27.700606518s" totalImagesPullingTime="19.395851546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:46.69794857 +0000 UTC m=+27.714969667" watchObservedRunningTime="2026-07-29 19:00:46.700606518 +0000 UTC m=+27.717627614" Jul 29 19:00:46.739943 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:46.739884 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr89q" podStartSLOduration=3.533694113 podStartE2EDuration="27.739870584s" totalImagesPullingTime="24.206176471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:46.739767238 +0000 UTC m=+27.756788333" watchObservedRunningTime="2026-07-29 19:00:46.739870584 +0000 UTC m=+27.756891677" Jul 29 19:00:47.543706 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:47.543683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:47.543706 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:47.543697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:47.544367 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:47.543790 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:47.544367 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:47.543869 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:47.677595 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:47.677490 2575 generic.go:356] "Generic (PLEG): container finished" podID="1815188e-0527-44ec-8aa3-c889efb12990" containerID="7839ad9a63a525311384a89d2515e60a526a75da439694cc2d0952f44a0af71a" exitCode=0 Jul 29 19:00:47.677953 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:47.677901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" event={"ID":"1815188e-0527-44ec-8aa3-c889efb12990","Type":"ContainerDied","Data":"7839ad9a63a525311384a89d2515e60a526a75da439694cc2d0952f44a0af71a"} Jul 29 19:00:47.678474 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:47.678451 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:47.699963 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:47.699934 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:00:48.455393 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:48.455199 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xgvm6"] Jul 29 19:00:48.455558 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:48.455530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:48.455680 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:48.455648 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:48.458032 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:48.458006 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nrdz9"] Jul 29 19:00:48.458141 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:48.458110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:48.458231 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:48.458206 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:48.458621 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:48.458601 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldxw2"] Jul 29 19:00:48.458701 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:48.458695 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:48.458789 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:48.458772 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:50.543748 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:50.543718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:50.544079 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:50.543757 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:50.544079 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:50.543757 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:50.544079 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:50.543831 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:50.544079 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:50.543944 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:50.544079 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:50.544049 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:50.686924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:50.686841 2575 generic.go:356] "Generic (PLEG): container finished" podID="1815188e-0527-44ec-8aa3-c889efb12990" containerID="230ee7387c5fe119509c0f5c70d9d1e3b66cb31874dce317e4b163b1e5f1bebd" exitCode=0 Jul 29 19:00:50.686924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:50.686889 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" event={"ID":"1815188e-0527-44ec-8aa3-c889efb12990","Type":"ContainerDied","Data":"230ee7387c5fe119509c0f5c70d9d1e3b66cb31874dce317e4b163b1e5f1bebd"} Jul 29 19:00:52.542935 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:52.542901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:52.542935 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:52.542936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:52.543449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:52.543019 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:52.543449 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:52.543041 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nrdz9" podUID="756f70ce-31b9-476f-a791-9365fdbe68ae" Jul 29 19:00:52.543449 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:52.543122 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:00:52.543449 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:52.543200 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xgvm6" podUID="40076bf8-469e-4e66-a9ba-85e5a8e876b5" Jul 29 19:00:53.123322 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.123289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:53.123509 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.123341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:53.123509 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:53.123448 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:53.123509 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:53.123456 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:53.123509 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:53.123504 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret podName:40076bf8-469e-4e66-a9ba-85e5a8e876b5 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:25.123485327 +0000 UTC m=+66.140506401 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret") pod "global-pull-secret-syncer-xgvm6" (UID: "40076bf8-469e-4e66-a9ba-85e5a8e876b5") : object "kube-system"/"original-pull-secret" not registered Jul 29 19:00:53.123749 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:53.123541 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:25.12353001 +0000 UTC m=+66.140551091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 19:00:53.224444 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.224413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:53.224604 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:53.224577 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 19:00:53.224604 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:53.224598 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 19:00:53.224668 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:53.224607 2575 projected.go:196] Error preparing data for projected volume kube-api-access-clchr for pod openshift-network-diagnostics/network-check-target-nrdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:53.224668 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:53.224655 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr podName:756f70ce-31b9-476f-a791-9365fdbe68ae nodeName:}" failed. No retries permitted until 2026-07-29 19:01:25.224642594 +0000 UTC m=+66.241663673 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-clchr" (UniqueName: "kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr") pod "network-check-target-nrdz9" (UID: "756f70ce-31b9-476f-a791-9365fdbe68ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 19:00:53.810803 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.810735 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeReady" Jul 29 19:00:53.811211 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.810870 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 19:00:53.849227 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.849193 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9"] Jul 29 19:00:53.870502 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.869733 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-758bf44b74-j7ths"] Jul 29 19:00:53.870502 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.869874 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9" Jul 29 19:00:53.872924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.872748 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 19:00:53.873064 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.872775 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 19:00:53.873064 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.872839 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dmlt7\"" Jul 29 19:00:53.881997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.881970 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-fmxls"] Jul 29 19:00:53.882136 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.882099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:53.885028 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.884887 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xdl87\"" Jul 29 19:00:53.885028 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.884915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 19:00:53.885028 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.884990 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 19:00:53.885028 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.884990 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 19:00:53.890104 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.889701 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 19:00:53.900176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.900151 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9"] Jul 29 19:00:53.900298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.900191 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-fmxls"] Jul 29 19:00:53.900298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.900213 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-758bf44b74-j7ths"] Jul 29 19:00:53.900298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.900236 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wpq2w"] Jul 29 19:00:53.900298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.900292 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:53.903065 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.903043 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 19:00:53.903184 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.903083 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 19:00:53.903184 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.903113 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mk4bt\"" Jul 29 19:00:53.916367 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.916347 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zljtc"] Jul 29 19:00:53.916491 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.916448 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wpq2w" Jul 29 19:00:53.918841 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.918822 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2wk54\"" Jul 29 19:00:53.918951 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.918844 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 19:00:53.918951 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.918871 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 19:00:53.918951 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.918933 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 19:00:53.926635 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.926617 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wpq2w"] Jul 29 19:00:53.926724 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.926657 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zljtc"] Jul 29 19:00:53.926783 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.926729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:53.928980 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.928957 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 19:00:53.929072 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.928993 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dnhc\"" Jul 29 19:00:53.929130 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:53.929117 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 19:00:54.029883 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.029846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-installation-pull-secrets\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.030132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.029919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-bound-sa-token\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.030132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.029944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxcz\" (UniqueName: \"kubernetes.io/projected/ba24905b-57b4-40fd-ad0f-4ca06010d4d5-kube-api-access-zxxcz\") pod \"ingress-canary-wpq2w\" (UID: \"ba24905b-57b4-40fd-ad0f-4ca06010d4d5\") " pod="openshift-ingress-canary/ingress-canary-wpq2w" Jul 29 19:00:54.030132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.029970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba24905b-57b4-40fd-ad0f-4ca06010d4d5-cert\") pod \"ingress-canary-wpq2w\" (UID: \"ba24905b-57b4-40fd-ad0f-4ca06010d4d5\") " pod="openshift-ingress-canary/ingress-canary-wpq2w" Jul 29 19:00:54.030132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.029997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-image-registry-private-configuration\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.030132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9sh\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-kube-api-access-zv9sh\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.030132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.030132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84f87992-9daa-4f03-853e-906e0268c139-config-volume\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.030132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54be0aba-6d57-43bb-a21e-0f458e2b38fd-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:54.030487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:54.030487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030199 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-trusted-ca\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.030487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjw9f\" (UniqueName: \"kubernetes.io/projected/b89afbf4-b7a0-4507-9616-67f0d86ac540-kube-api-access-pjw9f\") pod \"network-check-source-7d88cb55d8-b27q9\" (UID: \"b89afbf4-b7a0-4507-9616-67f0d86ac540\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9" Jul 29 19:00:54.030487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f87992-9daa-4f03-853e-906e0268c139-tmp-dir\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.030487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-certificates\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.030487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030324 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhwl\" (UniqueName: \"kubernetes.io/projected/84f87992-9daa-4f03-853e-906e0268c139-kube-api-access-wzhwl\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.030487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.030487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.030374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935916f1-b02b-4ae4-965a-7a6e508daf22-ca-trust-extracted\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.131597 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.131562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-certificates\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.131793 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.131606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhwl\" (UniqueName: \"kubernetes.io/projected/84f87992-9daa-4f03-853e-906e0268c139-kube-api-access-wzhwl\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.131793 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.131739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.131909 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.131801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935916f1-b02b-4ae4-965a-7a6e508daf22-ca-trust-extracted\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.133045 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.132497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-installation-pull-secrets\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.133045 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.132589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-certificates\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.133045 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.132619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-bound-sa-token\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.133045 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.132905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935916f1-b02b-4ae4-965a-7a6e508daf22-ca-trust-extracted\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.133045 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.132599 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:00:54.133045 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.132945 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:00:54.133045 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.133006 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:54.632987194 +0000 UTC m=+35.650008270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:00:54.133529 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxcz\" (UniqueName: \"kubernetes.io/projected/ba24905b-57b4-40fd-ad0f-4ca06010d4d5-kube-api-access-zxxcz\") pod \"ingress-canary-wpq2w\" (UID: \"ba24905b-57b4-40fd-ad0f-4ca06010d4d5\") " pod="openshift-ingress-canary/ingress-canary-wpq2w" Jul 29 19:00:54.133529 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba24905b-57b4-40fd-ad0f-4ca06010d4d5-cert\") pod \"ingress-canary-wpq2w\" (UID: \"ba24905b-57b4-40fd-ad0f-4ca06010d4d5\") " pod="openshift-ingress-canary/ingress-canary-wpq2w" Jul 29 19:00:54.133529 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-image-registry-private-configuration\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.133529 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9sh\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-kube-api-access-zv9sh\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.133529 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.133529 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84f87992-9daa-4f03-853e-906e0268c139-config-volume\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.133529 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54be0aba-6d57-43bb-a21e-0f458e2b38fd-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:54.133861 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:54.133861 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.133641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-trusted-ca\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.135837 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.135258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54be0aba-6d57-43bb-a21e-0f458e2b38fd-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:54.135837 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.135368 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:00:54.135837 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.135435 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:54.635416784 +0000 UTC m=+35.652437858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:00:54.135837 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.135469 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:00:54.135837 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.135580 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:00:54.635558534 +0000 UTC m=+35.652579620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:00:54.136145 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.136003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84f87992-9daa-4f03-853e-906e0268c139-config-volume\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.136195 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.136138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjw9f\" (UniqueName: \"kubernetes.io/projected/b89afbf4-b7a0-4507-9616-67f0d86ac540-kube-api-access-pjw9f\") pod \"network-check-source-7d88cb55d8-b27q9\" (UID: \"b89afbf4-b7a0-4507-9616-67f0d86ac540\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9" Jul 29 19:00:54.136245 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.136189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f87992-9daa-4f03-853e-906e0268c139-tmp-dir\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.136611 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.136593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f87992-9daa-4f03-853e-906e0268c139-tmp-dir\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.137472 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.137447 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-trusted-ca\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.140385 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.140352 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba24905b-57b4-40fd-ad0f-4ca06010d4d5-cert\") pod \"ingress-canary-wpq2w\" (UID: \"ba24905b-57b4-40fd-ad0f-4ca06010d4d5\") " pod="openshift-ingress-canary/ingress-canary-wpq2w" Jul 29 19:00:54.140786 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.140763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-installation-pull-secrets\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.140879 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.140832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-image-registry-private-configuration\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.141792 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.141768 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhwl\" (UniqueName: \"kubernetes.io/projected/84f87992-9daa-4f03-853e-906e0268c139-kube-api-access-wzhwl\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.142292 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.142271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-bound-sa-token\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.143091 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.143069 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9sh\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-kube-api-access-zv9sh\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.143739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.143718 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxcz\" (UniqueName: \"kubernetes.io/projected/ba24905b-57b4-40fd-ad0f-4ca06010d4d5-kube-api-access-zxxcz\") pod \"ingress-canary-wpq2w\" (UID: \"ba24905b-57b4-40fd-ad0f-4ca06010d4d5\") " pod="openshift-ingress-canary/ingress-canary-wpq2w" Jul 29 19:00:54.143913 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.143891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjw9f\" (UniqueName: \"kubernetes.io/projected/b89afbf4-b7a0-4507-9616-67f0d86ac540-kube-api-access-pjw9f\") pod \"network-check-source-7d88cb55d8-b27q9\" (UID: \"b89afbf4-b7a0-4507-9616-67f0d86ac540\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9" Jul 29 19:00:54.181573 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.181540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9" Jul 29 19:00:54.226055 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.226028 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wpq2w" Jul 29 19:00:54.543756 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.543680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:00:54.544024 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.543679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:00:54.544024 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.543681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:00:54.546646 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.546623 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 19:00:54.546646 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.546643 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qh2tz\"" Jul 29 19:00:54.546852 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.546632 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qs6c5\"" Jul 29 19:00:54.546852 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.546632 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 19:00:54.639836 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.639785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:54.640018 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.639892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:54.640018 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:54.639925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:54.640018 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.639965 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:00:54.640018 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.639989 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:00:54.640194 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.640018 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:00:54.640194 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.640043 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:00:54.640194 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.640062 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:55.640040781 +0000 UTC m=+36.657061857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:00:54.640194 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.640086 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:55.640068924 +0000 UTC m=+36.657090000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:00:54.640194 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:54.640114 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:00:55.640102097 +0000 UTC m=+36.657123170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:00:55.018421 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.018392 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wpq2w"] Jul 29 19:00:55.021542 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.021505 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9"] Jul 29 19:00:55.145711 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:55.145669 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba24905b_57b4_40fd_ad0f_4ca06010d4d5.slice/crio-d5c173c7c4bc2259e1980190cca1ea6a75eed0c9077bdabf8c6061c2f05ec90a WatchSource:0}: Error finding container d5c173c7c4bc2259e1980190cca1ea6a75eed0c9077bdabf8c6061c2f05ec90a: Status 404 returned error can't find the container with id d5c173c7c4bc2259e1980190cca1ea6a75eed0c9077bdabf8c6061c2f05ec90a Jul 29 19:00:55.146318 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:00:55.146295 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89afbf4_b7a0_4507_9616_67f0d86ac540.slice/crio-0dd4ca7353f4a93577a4ecb51e67c3332f8b9606cdf510e4e6b3a1315266d675 WatchSource:0}: Error finding container 0dd4ca7353f4a93577a4ecb51e67c3332f8b9606cdf510e4e6b3a1315266d675: Status 404 returned error can't find the container with id 0dd4ca7353f4a93577a4ecb51e67c3332f8b9606cdf510e4e6b3a1315266d675 Jul 29 19:00:55.647236 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.647046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:55.647400 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.647270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:55.647400 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:55.647210 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:00:55.647400 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:55.647313 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:00:55.647400 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:55.647365 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:00:55.647400 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:55.647371 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:00:55.647400 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.647298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:55.647400 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:55.647375 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:57.647356838 +0000 UTC m=+38.664377917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:00:55.647711 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:55.647433 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:00:57.647410728 +0000 UTC m=+38.664431805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:00:55.647711 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:55.647463 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:00:57.647450832 +0000 UTC m=+38.664471921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:00:55.701797 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.701760 2575 generic.go:356] "Generic (PLEG): container finished" podID="1815188e-0527-44ec-8aa3-c889efb12990" containerID="b9aa705abed1f4675ca4c82b397821b17a02a488501ca744b5337497c4e8919e" exitCode=0 Jul 29 19:00:55.701975 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.701839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" event={"ID":"1815188e-0527-44ec-8aa3-c889efb12990","Type":"ContainerDied","Data":"b9aa705abed1f4675ca4c82b397821b17a02a488501ca744b5337497c4e8919e"} Jul 29 19:00:55.703056 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.703021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9" event={"ID":"b89afbf4-b7a0-4507-9616-67f0d86ac540","Type":"ContainerStarted","Data":"0dd4ca7353f4a93577a4ecb51e67c3332f8b9606cdf510e4e6b3a1315266d675"} Jul 29 19:00:55.704039 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:55.704021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wpq2w" event={"ID":"ba24905b-57b4-40fd-ad0f-4ca06010d4d5","Type":"ContainerStarted","Data":"d5c173c7c4bc2259e1980190cca1ea6a75eed0c9077bdabf8c6061c2f05ec90a"} Jul 29 19:00:56.714364 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:56.713674 2575 generic.go:356] "Generic (PLEG): container finished" podID="1815188e-0527-44ec-8aa3-c889efb12990" containerID="16607ced7e8e20abd267fe487bfefe1c098d65c4f49baab95a962c20746a655a" exitCode=0 Jul 29 19:00:56.714364 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:56.713721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" event={"ID":"1815188e-0527-44ec-8aa3-c889efb12990","Type":"ContainerDied","Data":"16607ced7e8e20abd267fe487bfefe1c098d65c4f49baab95a962c20746a655a"} Jul 29 19:00:57.666047 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:57.666001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:00:57.666256 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:57.666084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:00:57.666256 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:57.666178 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:00:57.666256 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:57.666202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:00:57.666256 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:57.666205 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:00:57.666256 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:57.666256 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:00:57.666660 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:57.666184 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:00:57.666660 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:57.666308 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:01.666291396 +0000 UTC m=+42.683312486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:00:57.666660 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:57.666326 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:01:01.666319166 +0000 UTC m=+42.683340240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:00:57.666660 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:00:57.666336 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:01.666331037 +0000 UTC m=+42.683352110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:00:57.720253 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:57.720215 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" event={"ID":"1815188e-0527-44ec-8aa3-c889efb12990","Type":"ContainerStarted","Data":"a82bbe6940ad15839682598162cdc6562e29eddd3636c881dc861d7400212917"} Jul 29 19:00:57.746361 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:57.746306 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4p7vd" podStartSLOduration=6.628224547 podStartE2EDuration="38.746286851s" totalImagesPullingTime="32.118062304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:57.743079635 +0000 UTC m=+38.760100732" watchObservedRunningTime="2026-07-29 19:00:57.746286851 +0000 UTC m=+38.763307947" Jul 29 19:00:59.727055 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:59.726964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9" event={"ID":"b89afbf4-b7a0-4507-9616-67f0d86ac540","Type":"ContainerStarted","Data":"bcae8ba5fd90cc73f269a8ebac6fd0ce8360496cdf0ad6f942eefb0c584c56a9"} Jul 29 19:00:59.728463 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:59.728442 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wpq2w" event={"ID":"ba24905b-57b4-40fd-ad0f-4ca06010d4d5","Type":"ContainerStarted","Data":"3a1099f780cd9120fe7ed99e12c7c548edc2d38fedc844e0408a985f078eb128"} Jul 29 19:00:59.743551 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:59.743376 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-b27q9" podStartSLOduration=28.516938989 podStartE2EDuration="32.743358462s" totalImagesPullingTime="4.226419473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:59.740616111 +0000 UTC m=+40.757637207" watchObservedRunningTime="2026-07-29 19:00:59.743358462 +0000 UTC m=+40.760379555" Jul 29 19:00:59.757531 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:00:59.757471 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wpq2w" podStartSLOduration=2.524575951 podStartE2EDuration="6.757457273s" totalImagesPullingTime="4.232881322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:00:59.757037094 +0000 UTC m=+40.774058191" watchObservedRunningTime="2026-07-29 19:00:59.757457273 +0000 UTC m=+40.774478348" Jul 29 19:01:01.702191 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:01.702148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:01.702216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:01.702239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:01.702326 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:01.702372 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:01.702344 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:01.702349 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:01.702433 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:09.70241558 +0000 UTC m=+50.719436660 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:01.702544 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:01:09.70252622 +0000 UTC m=+50.719547298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:01:01.702700 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:01.702562 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:09.702555707 +0000 UTC m=+50.719576780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:01:09.762805 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:09.762769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:09.762816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:09.762866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:09.762912 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:09.762947 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:09.762957 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:09.762970 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:25.762954808 +0000 UTC m=+66.779975881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:09.762974 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:09.762999 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:25.762989364 +0000 UTC m=+66.780010441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:01:09.763179 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:09.763030 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:01:25.763013565 +0000 UTC m=+66.780034639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:01:19.692632 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:19.692601 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmb68" Jul 29 19:01:25.171763 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.171722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:01:25.172198 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.171796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:01:25.174828 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.174807 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 19:01:25.174828 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.174821 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 19:01:25.182112 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.182093 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 19:01:25.182216 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.182173 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:02:29.182152208 +0000 UTC m=+130.199173284 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : secret "metrics-daemon-secret" not found Jul 29 19:01:25.185418 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.185379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40076bf8-469e-4e66-a9ba-85e5a8e876b5-original-pull-secret\") pod \"global-pull-secret-syncer-xgvm6\" (UID: \"40076bf8-469e-4e66-a9ba-85e5a8e876b5\") " pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:01:25.272252 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.272215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:01:25.274669 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.274647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clchr\" (UniqueName: \"kubernetes.io/projected/756f70ce-31b9-476f-a791-9365fdbe68ae-kube-api-access-clchr\") pod \"network-check-target-nrdz9\" (UID: \"756f70ce-31b9-476f-a791-9365fdbe68ae\") " pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:01:25.462405 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.462322 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xgvm6" Jul 29 19:01:25.471660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.471633 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qs6c5\"" Jul 29 19:01:25.478806 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.478785 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:01:25.630195 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.630164 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xgvm6"] Jul 29 19:01:25.633544 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:01:25.633503 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40076bf8_469e_4e66_a9ba_85e5a8e876b5.slice/crio-9a765a31d80126b0b9e2e0b27994af2ccd8b13d2811d176d071bc2b1742a5392 WatchSource:0}: Error finding container 9a765a31d80126b0b9e2e0b27994af2ccd8b13d2811d176d071bc2b1742a5392: Status 404 returned error can't find the container with id 9a765a31d80126b0b9e2e0b27994af2ccd8b13d2811d176d071bc2b1742a5392 Jul 29 19:01:25.646277 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.646249 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nrdz9"] Jul 29 19:01:25.649124 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:01:25.649098 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756f70ce_31b9_476f_a791_9365fdbe68ae.slice/crio-0d8baa0c01e7f3fdf2e9f75240dd28b5aed5301323c78fe577cec11074154acd WatchSource:0}: Error finding container 0d8baa0c01e7f3fdf2e9f75240dd28b5aed5301323c78fe577cec11074154acd: Status 404 returned error can't find the container with id 0d8baa0c01e7f3fdf2e9f75240dd28b5aed5301323c78fe577cec11074154acd Jul 29 19:01:25.776687 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.776650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:01:25.776823 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.776700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:01:25.776823 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.776755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:01:25.776823 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.776797 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:01:25.776973 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.776843 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:01:25.776973 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.776857 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:01:25.776973 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.776863 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:57.776844843 +0000 UTC m=+98.793865934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:01:25.776973 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.776871 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:01:25.776973 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.776883 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:01:57.776871627 +0000 UTC m=+98.793892705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:01:25.776973 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:25.776917 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:01:57.776901856 +0000 UTC m=+98.793922931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:01:25.791930 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.791897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nrdz9" event={"ID":"756f70ce-31b9-476f-a791-9365fdbe68ae","Type":"ContainerStarted","Data":"e033b49eada100c2f67db54c31221c8b8d0bd03591a329ccc9ffce9de0261742"} Jul 29 19:01:25.792075 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.792056 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:01:25.792148 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.792101 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nrdz9" event={"ID":"756f70ce-31b9-476f-a791-9365fdbe68ae","Type":"ContainerStarted","Data":"0d8baa0c01e7f3fdf2e9f75240dd28b5aed5301323c78fe577cec11074154acd"} Jul 29 19:01:25.792958 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.792936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xgvm6" event={"ID":"40076bf8-469e-4e66-a9ba-85e5a8e876b5","Type":"ContainerStarted","Data":"9a765a31d80126b0b9e2e0b27994af2ccd8b13d2811d176d071bc2b1742a5392"} Jul 29 19:01:25.808637 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:25.807165 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nrdz9" podStartSLOduration=66.807140508 podStartE2EDuration="1m6.807140508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:01:25.805643598 +0000 UTC m=+66.822664697" watchObservedRunningTime="2026-07-29 19:01:25.807140508 +0000 UTC m=+66.824161606" Jul 29 19:01:30.807041 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:30.806949 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xgvm6" event={"ID":"40076bf8-469e-4e66-a9ba-85e5a8e876b5","Type":"ContainerStarted","Data":"fbe2e88c54d228442ac358f756a9b6f0f269e74c014f2ab061ca5cd899ac3541"} Jul 29 19:01:30.822528 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:30.822477 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xgvm6" podStartSLOduration=66.938350627 podStartE2EDuration="1m11.822462123s" totalImagesPullingTime="4.884111496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:01:30.821191602 +0000 UTC m=+71.838212677" watchObservedRunningTime="2026-07-29 19:01:30.822462123 +0000 UTC m=+71.839483219" Jul 29 19:01:56.798214 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:56.798083 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nrdz9" Jul 29 19:01:57.808677 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:57.808625 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:57.808704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:01:57.808745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:57.808781 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:57.808800 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:57.808841 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:57.808876 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:03:01.808858979 +0000 UTC m=+162.825880060 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:57.808906 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:03:01.80889404 +0000 UTC m=+162.825915113 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:57.808848 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:01:57.809054 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:01:57.808949 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:03:01.808942022 +0000 UTC m=+162.825963096 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:02:06.895939 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.895902 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl"] Jul 29 19:02:06.899100 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.899084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:06.902752 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.902733 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 19:02:06.902859 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.902733 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 19:02:06.903051 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.903030 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 19:02:06.903196 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.903067 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 19:02:06.903196 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.903078 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 19:02:06.903196 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.903120 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 19:02:06.903196 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.903070 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 19:02:06.910101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.910079 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl"] Jul 29 19:02:06.972621 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.972588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-ca\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:06.972621 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.972620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:06.972840 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.972654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:06.972840 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.972679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-hub\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:06.972840 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.972719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:06.972840 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:06.972827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44pl\" (UniqueName: \"kubernetes.io/projected/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-kube-api-access-q44pl\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.073378 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.073345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-ca\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.073378 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.073379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.073603 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.073410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.073603 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.073436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-hub\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.073603 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.073570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.073733 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.073631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44pl\" (UniqueName: \"kubernetes.io/projected/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-kube-api-access-q44pl\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.074217 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.074195 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.075979 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.075949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-ca\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.076096 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.075999 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.076374 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.076354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-hub\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.076454 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.076434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.082254 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.082235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44pl\" (UniqueName: \"kubernetes.io/projected/78cbea3f-8027-4406-aa1f-cb3ba988d9ae-kube-api-access-q44pl\") pod \"cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl\" (UID: \"78cbea3f-8027-4406-aa1f-cb3ba988d9ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.228180 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.228069 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:02:07.353428 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.353395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl"] Jul 29 19:02:07.356772 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:02:07.356738 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78cbea3f_8027_4406_aa1f_cb3ba988d9ae.slice/crio-09702e6783c820b8a93effccfb6a13e686c6d8cc78b7bd4d03461b80bc803be5 WatchSource:0}: Error finding container 09702e6783c820b8a93effccfb6a13e686c6d8cc78b7bd4d03461b80bc803be5: Status 404 returned error can't find the container with id 09702e6783c820b8a93effccfb6a13e686c6d8cc78b7bd4d03461b80bc803be5 Jul 29 19:02:07.893619 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:07.893575 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" event={"ID":"78cbea3f-8027-4406-aa1f-cb3ba988d9ae","Type":"ContainerStarted","Data":"09702e6783c820b8a93effccfb6a13e686c6d8cc78b7bd4d03461b80bc803be5"} Jul 29 19:02:11.908187 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:11.908125 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" event={"ID":"78cbea3f-8027-4406-aa1f-cb3ba988d9ae","Type":"ContainerStarted","Data":"f6a7ee75b460393e0fd5a5ae444b88a9483af4194204377c97767d81a57a2ea8"} Jul 29 19:02:11.908608 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:11.908229 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" event={"ID":"78cbea3f-8027-4406-aa1f-cb3ba988d9ae","Type":"ContainerStarted","Data":"22f641f7157592a5079050019528183d55977a991c4743d6fcfac86549ef2674"} Jul 29 19:02:11.908608 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:11.908534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" event={"ID":"78cbea3f-8027-4406-aa1f-cb3ba988d9ae","Type":"ContainerStarted","Data":"c4afd57f9bace6248037f70e54fe0cfd7a446c895dd8b933df9c665df09b064b"} Jul 29 19:02:11.929226 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:11.929180 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" podStartSLOduration=2.344107091 podStartE2EDuration="5.929151612s" totalImagesPullingTime="3.585044521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:11.928196381 +0000 UTC m=+112.945217476" watchObservedRunningTime="2026-07-29 19:02:11.929151612 +0000 UTC m=+112.946172690" Jul 29 19:02:29.247502 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:29.247458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:02:29.248016 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:02:29.247626 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 19:02:29.248016 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:02:29.247703 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:31.247686013 +0000 UTC m=+252.264707087 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : secret "metrics-daemon-secret" not found Jul 29 19:02:56.892530 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:02:56.892474 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" podUID="935916f1-b02b-4ae4-965a-7a6e508daf22" Jul 29 19:02:56.910592 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:02:56.910566 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" podUID="54be0aba-6d57-43bb-a21e-0f458e2b38fd" Jul 29 19:02:56.936867 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:02:56.936833 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zljtc" podUID="84f87992-9daa-4f03-853e-906e0268c139" Jul 29 19:02:57.037063 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:57.037036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:02:57.037203 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:02:57.037036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:02:57.555741 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:02:57.555706 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:03:01.882409 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:03:01.882366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:03:01.882432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:03:01.882462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:03:01.882572 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:03:01.882597 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:03:01.882610 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758bf44b74-j7ths: secret "image-registry-tls" not found Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:03:01.882581 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:03:01.882665 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls podName:84f87992-9daa-4f03-853e-906e0268c139 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:03.882650205 +0000 UTC m=+284.899671284 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls") pod "dns-default-zljtc" (UID: "84f87992-9daa-4f03-853e-906e0268c139") : secret "dns-default-metrics-tls" not found Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:03:01.882680 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls podName:935916f1-b02b-4ae4-965a-7a6e508daf22 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:03.882673876 +0000 UTC m=+284.899694950 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls") pod "image-registry-758bf44b74-j7ths" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22") : secret "image-registry-tls" not found Jul 29 19:03:01.882992 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:03:01.882690 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert podName:54be0aba-6d57-43bb-a21e-0f458e2b38fd nodeName:}" failed. No retries permitted until 2026-07-29 19:05:03.88268497 +0000 UTC m=+284.899706043 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fmxls" (UID: "54be0aba-6d57-43bb-a21e-0f458e2b38fd") : secret "networking-console-plugin-cert" not found Jul 29 19:03:07.543559 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:03:07.543499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zljtc" Jul 29 19:03:12.543691 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:03:12.543588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:04:31.327574 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:31.327534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:04:31.328042 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:31.327637 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 19:04:31.328042 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:31.327698 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs podName:11505d8a-ce7a-4382-bdf7-9d77110b4ef0 nodeName:}" failed. No retries permitted until 2026-07-29 19:06:33.327681177 +0000 UTC m=+374.344702262 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs") pod "network-metrics-daemon-ldxw2" (UID: "11505d8a-ce7a-4382-bdf7-9d77110b4ef0") : secret "metrics-daemon-secret" not found Jul 29 19:04:45.351735 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.351655 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-llzvq"] Jul 29 19:04:45.353693 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.353676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.356157 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.356137 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 19:04:45.356245 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.356185 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 19:04:45.356981 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.356959 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 19:04:45.357680 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.357659 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 19:04:45.357772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.357722 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4lxqb\"" Jul 29 19:04:45.365982 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.365962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-llzvq"] Jul 29 19:04:45.428654 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.428619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.428654 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.428655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88btc\" (UniqueName: \"kubernetes.io/projected/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-api-access-88btc\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.428864 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.428677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ebd00696-c986-47ae-a7d3-34469a66ed98-crio-socket\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.428864 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.428786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.428864 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.428842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebd00696-c986-47ae-a7d3-34469a66ed98-data-volume\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.530082 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.530051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.530188 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.530084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88btc\" (UniqueName: \"kubernetes.io/projected/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-api-access-88btc\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.530188 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.530105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ebd00696-c986-47ae-a7d3-34469a66ed98-crio-socket\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.530188 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.530154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.530297 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:45.530186 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 19:04:45.530297 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.530198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebd00696-c986-47ae-a7d3-34469a66ed98-data-volume\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.530297 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:45.530260 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:46.030240268 +0000 UTC m=+267.047261349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : secret "insights-runtime-extractor-tls" not found Jul 29 19:04:45.530297 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.530273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ebd00696-c986-47ae-a7d3-34469a66ed98-crio-socket\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.530297 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:45.530284 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 19:04:45.530457 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:45.530361 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:46.030346061 +0000 UTC m=+267.047367142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : configmap "kube-rbac-proxy" not found Jul 29 19:04:45.530568 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.530554 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebd00696-c986-47ae-a7d3-34469a66ed98-data-volume\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:45.538413 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:45.538385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88btc\" (UniqueName: \"kubernetes.io/projected/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-api-access-88btc\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:46.034218 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:46.034160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:46.034218 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:46.034236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:46.034486 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:46.034297 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 19:04:46.034486 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:46.034341 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 19:04:46.034486 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:46.034349 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:47.034335201 +0000 UTC m=+268.051356278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : configmap "kube-rbac-proxy" not found Jul 29 19:04:46.034486 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:46.034460 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:47.034438999 +0000 UTC m=+268.051460085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : secret "insights-runtime-extractor-tls" not found Jul 29 19:04:47.041623 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:47.041585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:47.042007 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:47.041645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:47.042007 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:47.041712 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 19:04:47.042007 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:47.041747 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 19:04:47.042007 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:47.041770 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:49.041755059 +0000 UTC m=+270.058776133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : configmap "kube-rbac-proxy" not found Jul 29 19:04:47.042007 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:47.041801 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:49.041789203 +0000 UTC m=+270.058810281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : secret "insights-runtime-extractor-tls" not found Jul 29 19:04:47.135374 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:47.135349 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4zjgp_b8eb708d-e411-4fc8-bca7-9781b260c738/dns-node-resolver/0.log" Jul 29 19:04:48.535501 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:48.535473 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qxrp2_7d1c8b71-e806-48a9-9037-5fe23acc92ec/node-ca/0.log" Jul 29 19:04:49.058418 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:49.058376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:49.058619 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:49.058479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:49.058619 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:49.058505 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 19:04:49.058619 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:49.058597 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:53.058581535 +0000 UTC m=+274.075602609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : configmap "kube-rbac-proxy" not found Jul 29 19:04:49.058619 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:49.058607 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 19:04:49.058762 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:49.058651 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:04:53.05863626 +0000 UTC m=+274.075657335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : secret "insights-runtime-extractor-tls" not found Jul 29 19:04:49.335495 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:49.335409 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wpq2w_ba24905b-57b4-40fd-ad0f-4ca06010d4d5/serve-healthcheck-canary/0.log" Jul 29 19:04:53.096206 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:53.096147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:53.096627 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:04:53.096267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:04:53.096627 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:53.096271 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 19:04:53.096627 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:53.096329 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:01.096314949 +0000 UTC m=+282.113336024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : configmap "kube-rbac-proxy" not found Jul 29 19:04:53.096627 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:53.096366 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 19:04:53.096627 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:04:53.096414 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls podName:ebd00696-c986-47ae-a7d3-34469a66ed98 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:01.096399855 +0000 UTC m=+282.113420929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llzvq" (UID: "ebd00696-c986-47ae-a7d3-34469a66ed98") : secret "insights-runtime-extractor-tls" not found Jul 29 19:05:00.037561 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:05:00.037511 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" podUID="54be0aba-6d57-43bb-a21e-0f458e2b38fd" Jul 29 19:05:00.037561 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:05:00.037529 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" podUID="935916f1-b02b-4ae4-965a-7a6e508daf22" Jul 29 19:05:00.382671 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:00.382642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:05:00.382828 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:00.382642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:05:01.166979 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:01.166937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:05:01.167363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:01.166996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:05:01.167467 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:01.167450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebd00696-c986-47ae-a7d3-34469a66ed98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:05:01.169312 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:01.169293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebd00696-c986-47ae-a7d3-34469a66ed98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llzvq\" (UID: \"ebd00696-c986-47ae-a7d3-34469a66ed98\") " pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:05:01.262199 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:01.262164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llzvq" Jul 29 19:05:01.385538 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:01.385492 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-llzvq"] Jul 29 19:05:01.388483 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:05:01.388457 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd00696_c986_47ae_a7d3_34469a66ed98.slice/crio-b50bb411ddd608a829b1ed79ce20d4ebbdff813a57219275ed2d11b6fe165f6d WatchSource:0}: Error finding container b50bb411ddd608a829b1ed79ce20d4ebbdff813a57219275ed2d11b6fe165f6d: Status 404 returned error can't find the container with id b50bb411ddd608a829b1ed79ce20d4ebbdff813a57219275ed2d11b6fe165f6d Jul 29 19:05:02.390673 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:02.390629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llzvq" event={"ID":"ebd00696-c986-47ae-a7d3-34469a66ed98","Type":"ContainerStarted","Data":"b50bb411ddd608a829b1ed79ce20d4ebbdff813a57219275ed2d11b6fe165f6d"} Jul 29 19:05:03.891301 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.891249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:05:03.891738 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.891317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:05:03.891738 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.891364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:05:03.893869 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.893833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84f87992-9daa-4f03-853e-906e0268c139-metrics-tls\") pod \"dns-default-zljtc\" (UID: \"84f87992-9daa-4f03-853e-906e0268c139\") " pod="openshift-dns/dns-default-zljtc" Jul 29 19:05:03.894220 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.894199 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/54be0aba-6d57-43bb-a21e-0f458e2b38fd-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fmxls\" (UID: \"54be0aba-6d57-43bb-a21e-0f458e2b38fd\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:05:03.894275 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.894238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"image-registry-758bf44b74-j7ths\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:05:03.947624 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.947594 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dnhc\"" Jul 29 19:05:03.955654 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.955615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zljtc" Jul 29 19:05:03.986268 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.986230 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xdl87\"" Jul 29 19:05:03.986437 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.986327 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mk4bt\"" Jul 29 19:05:03.993788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.993758 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:05:03.993969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:03.993763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" Jul 29 19:05:06.443208 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:06.443179 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zljtc"] Jul 29 19:05:06.445989 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:05:06.445956 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f87992_9daa_4f03_853e_906e0268c139.slice/crio-1135299d2259721fdcabb073dbdbd8eb9890f9e127d2cf86bda9ddc165d42ea4 WatchSource:0}: Error finding container 1135299d2259721fdcabb073dbdbd8eb9890f9e127d2cf86bda9ddc165d42ea4: Status 404 returned error can't find the container with id 1135299d2259721fdcabb073dbdbd8eb9890f9e127d2cf86bda9ddc165d42ea4 Jul 29 19:05:06.461588 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:06.461507 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-758bf44b74-j7ths"] Jul 29 19:05:06.464404 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:05:06.464375 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935916f1_b02b_4ae4_965a_7a6e508daf22.slice/crio-e769908ce1371bea05e69016b1b6515743ab3a41b4353224621a01a0e80487d8 WatchSource:0}: Error finding container e769908ce1371bea05e69016b1b6515743ab3a41b4353224621a01a0e80487d8: Status 404 returned error can't find the container with id e769908ce1371bea05e69016b1b6515743ab3a41b4353224621a01a0e80487d8 Jul 29 19:05:06.661491 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:06.661457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-fmxls"] Jul 29 19:05:06.664697 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:05:06.664661 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54be0aba_6d57_43bb_a21e_0f458e2b38fd.slice/crio-fb15f2dd8356b06a167c1a5392b8c7f879f1b6e827ac343eff88dd95271c5de4 WatchSource:0}: Error finding container fb15f2dd8356b06a167c1a5392b8c7f879f1b6e827ac343eff88dd95271c5de4: Status 404 returned error can't find the container with id fb15f2dd8356b06a167c1a5392b8c7f879f1b6e827ac343eff88dd95271c5de4 Jul 29 19:05:07.414533 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:07.414477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llzvq" event={"ID":"ebd00696-c986-47ae-a7d3-34469a66ed98","Type":"ContainerStarted","Data":"0a31e4c3531e00de683e88b140a6b8dd6673281459c754af785859056fa11005"} Jul 29 19:05:07.416239 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:07.416187 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" event={"ID":"54be0aba-6d57-43bb-a21e-0f458e2b38fd","Type":"ContainerStarted","Data":"fb15f2dd8356b06a167c1a5392b8c7f879f1b6e827ac343eff88dd95271c5de4"} Jul 29 19:05:07.418761 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:07.418731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" event={"ID":"935916f1-b02b-4ae4-965a-7a6e508daf22","Type":"ContainerStarted","Data":"71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf"} Jul 29 19:05:07.418891 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:07.418781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" event={"ID":"935916f1-b02b-4ae4-965a-7a6e508daf22","Type":"ContainerStarted","Data":"e769908ce1371bea05e69016b1b6515743ab3a41b4353224621a01a0e80487d8"} Jul 29 19:05:07.418952 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:07.418925 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:05:07.420279 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:07.420227 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zljtc" event={"ID":"84f87992-9daa-4f03-853e-906e0268c139","Type":"ContainerStarted","Data":"1135299d2259721fdcabb073dbdbd8eb9890f9e127d2cf86bda9ddc165d42ea4"} Jul 29 19:05:07.441980 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:07.441923 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" podStartSLOduration=288.441905864 podStartE2EDuration="4m48.441905864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:07.441108042 +0000 UTC m=+288.458129138" watchObservedRunningTime="2026-07-29 19:05:07.441905864 +0000 UTC m=+288.458926962" Jul 29 19:05:11.436432 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:11.436391 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llzvq" event={"ID":"ebd00696-c986-47ae-a7d3-34469a66ed98","Type":"ContainerStarted","Data":"6b8a462f632a228f2caf001626e3abba7c3a189f1fb97d1cf0e3772f83723149"} Jul 29 19:05:11.438119 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:11.438088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" event={"ID":"54be0aba-6d57-43bb-a21e-0f458e2b38fd","Type":"ContainerStarted","Data":"7abae9b54a36449b2aa66493633207318263e85b51b53e901a29a9ba1e5c90d8"} Jul 29 19:05:11.439933 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:11.439904 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zljtc" event={"ID":"84f87992-9daa-4f03-853e-906e0268c139","Type":"ContainerStarted","Data":"f02a37a67836218b94c3e2395561a8331a6583df9cdb7c3e19f8ba32417be138"} Jul 29 19:05:11.440023 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:11.439953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zljtc" event={"ID":"84f87992-9daa-4f03-853e-906e0268c139","Type":"ContainerStarted","Data":"4b4a87b2ab4cee3795b105e257681f06e6e56a54c430518e8f929f0c9f5d6e95"} Jul 29 19:05:11.440083 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:11.440044 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zljtc" Jul 29 19:05:11.452937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:11.452889 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fmxls" podStartSLOduration=256.382507819 podStartE2EDuration="4m20.452873405s" totalImagesPullingTime="4.070365586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:11.452544147 +0000 UTC m=+292.469565242" watchObservedRunningTime="2026-07-29 19:05:11.452873405 +0000 UTC m=+292.469894502" Jul 29 19:05:11.469126 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:11.469079 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zljtc" podStartSLOduration=254.49995932 podStartE2EDuration="4m18.469063142s" totalImagesPullingTime="3.969103822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:11.468276249 +0000 UTC m=+292.485297369" watchObservedRunningTime="2026-07-29 19:05:11.469063142 +0000 UTC m=+292.486084237" Jul 29 19:05:14.452956 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:14.452906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llzvq" event={"ID":"ebd00696-c986-47ae-a7d3-34469a66ed98","Type":"ContainerStarted","Data":"2c202e0f9162e761afbabe60c42a0c3926918fc30f36a3624fea579a71fa36b8"} Jul 29 19:05:14.470397 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:14.470345 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-llzvq" podStartSLOduration=16.705867037 podStartE2EDuration="29.470330021s" totalImagesPullingTime="12.764462984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:04:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:14.46847292 +0000 UTC m=+295.485494015" watchObservedRunningTime="2026-07-29 19:05:14.470330021 +0000 UTC m=+295.487351117" Jul 29 19:05:15.544554 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:05:15.544484 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ldxw2" podUID="11505d8a-ce7a-4382-bdf7-9d77110b4ef0" Jul 29 19:05:17.230259 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:17.230212 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" podUID="78cbea3f-8027-4406-aa1f-cb3ba988d9ae" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 19:05:19.371092 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:19.371062 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 19:05:21.446403 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:21.446371 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zljtc" Jul 29 19:05:23.664061 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.664025 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-chrfj"] Jul 29 19:05:23.669725 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.669703 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.672694 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.672663 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 19:05:23.672694 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.672689 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 19:05:23.673628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.673607 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 19:05:23.673628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.673616 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 19:05:23.673806 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.673641 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 19:05:23.673806 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.673607 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-58xhx\"" Jul 29 19:05:23.673806 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.673682 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 19:05:23.734724 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-wtmp\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.734724 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnfx\" (UniqueName: \"kubernetes.io/projected/833b3fa9-8636-4eda-a9d4-543a83510835-kube-api-access-kfnfx\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.734957 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-sys\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.734957 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.734957 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-tls\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.734957 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-root\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.734957 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-accelerators-collector-config\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.734957 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-textfile\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.734957 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.734905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/833b3fa9-8636-4eda-a9d4-543a83510835-metrics-client-ca\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835353 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-wtmp\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835547 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnfx\" (UniqueName: \"kubernetes.io/projected/833b3fa9-8636-4eda-a9d4-543a83510835-kube-api-access-kfnfx\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835547 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-sys\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835547 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835547 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-tls\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835547 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-root\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835547 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835511 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-wtmp\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835547 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-accelerators-collector-config\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835867 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835580 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-root\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835867 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-textfile\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835867 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/833b3fa9-8636-4eda-a9d4-543a83510835-sys\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835867 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.835627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/833b3fa9-8636-4eda-a9d4-543a83510835-metrics-client-ca\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.835867 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:05:23.835631 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 19:05:23.835867 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:05:23.835693 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-tls podName:833b3fa9-8636-4eda-a9d4-543a83510835 nodeName:}" failed. No retries permitted until 2026-07-29 19:05:24.335673386 +0000 UTC m=+305.352694463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-tls") pod "node-exporter-chrfj" (UID: "833b3fa9-8636-4eda-a9d4-543a83510835") : secret "node-exporter-tls" not found Jul 29 19:05:23.836169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.836123 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-textfile\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.836228 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.836190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/833b3fa9-8636-4eda-a9d4-543a83510835-metrics-client-ca\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.836279 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.836235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-accelerators-collector-config\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.837901 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.837881 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.845245 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.845222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnfx\" (UniqueName: \"kubernetes.io/projected/833b3fa9-8636-4eda-a9d4-543a83510835-kube-api-access-kfnfx\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:23.998108 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.998018 2575 patch_prober.go:28] interesting pod/image-registry-758bf44b74-j7ths container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 19:05:23.998108 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:23.998090 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" podUID="935916f1-b02b-4ae4-965a-7a6e508daf22" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:05:24.340093 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.339994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-tls\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:24.342292 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.342269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/833b3fa9-8636-4eda-a9d4-543a83510835-node-exporter-tls\") pod \"node-exporter-chrfj\" (UID: \"833b3fa9-8636-4eda-a9d4-543a83510835\") " pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:24.579137 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.579103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chrfj" Jul 29 19:05:24.590851 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:05:24.590787 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833b3fa9_8636_4eda_a9d4_543a83510835.slice/crio-f08688b1e50e31fbde4fc5434cb1294ecd7b91d816facef9c8c083e7931d24ad WatchSource:0}: Error finding container f08688b1e50e31fbde4fc5434cb1294ecd7b91d816facef9c8c083e7931d24ad: Status 404 returned error can't find the container with id f08688b1e50e31fbde4fc5434cb1294ecd7b91d816facef9c8c083e7931d24ad Jul 29 19:05:24.592865 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.592834 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:05:24.745348 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.745305 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 19:05:24.752115 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.752088 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.754371 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.754347 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 19:05:24.754746 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.754726 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 19:05:24.754746 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.754735 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 19:05:24.754898 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.754743 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 19:05:24.754898 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.754851 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 19:05:24.755006 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.754951 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 19:05:24.755165 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.755144 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 19:05:24.755269 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.755166 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jgfdw\"" Jul 29 19:05:24.755269 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.755173 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 19:05:24.755269 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.755151 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 19:05:24.762628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.762607 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 19:05:24.846324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/569c6b31-d722-4187-868b-bc4a6f019c44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/569c6b31-d722-4187-868b-bc4a6f019c44-config-out\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846563 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569c6b31-d722-4187-868b-bc4a6f019c44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846563 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846563 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846563 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846563 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/569c6b31-d722-4187-868b-bc4a6f019c44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846563 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-web-config\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846563 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846543 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846768 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvl65\" (UniqueName: \"kubernetes.io/projected/569c6b31-d722-4187-868b-bc4a6f019c44-kube-api-access-bvl65\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846768 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/569c6b31-d722-4187-868b-bc4a6f019c44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.846768 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.846604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-config-volume\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.947898 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.947866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvl65\" (UniqueName: \"kubernetes.io/projected/569c6b31-d722-4187-868b-bc4a6f019c44-kube-api-access-bvl65\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.947898 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.947906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/569c6b31-d722-4187-868b-bc4a6f019c44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948105 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.947922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-config-volume\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948105 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948105 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/569c6b31-d722-4187-868b-bc4a6f019c44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948244 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/569c6b31-d722-4187-868b-bc4a6f019c44-config-out\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948377 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569c6b31-d722-4187-868b-bc4a6f019c44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948557 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948652 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948710 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948762 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/569c6b31-d722-4187-868b-bc4a6f019c44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948815 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-web-config\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.948815 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.948792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.949052 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.949029 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/569c6b31-d722-4187-868b-bc4a6f019c44-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.949427 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.949270 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/569c6b31-d722-4187-868b-bc4a6f019c44-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.950568 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.950544 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569c6b31-d722-4187-868b-bc4a6f019c44-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.951102 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.951075 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.951193 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.951120 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/569c6b31-d722-4187-868b-bc4a6f019c44-config-out\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.951562 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.951538 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/569c6b31-d722-4187-868b-bc4a6f019c44-tls-assets\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.951562 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.951563 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-config-volume\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.952209 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.951886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.952209 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.952162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.952209 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.952167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.952871 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.952850 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.953123 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.953106 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/569c6b31-d722-4187-868b-bc4a6f019c44-web-config\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:24.957385 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:24.957366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvl65\" (UniqueName: \"kubernetes.io/projected/569c6b31-d722-4187-868b-bc4a6f019c44-kube-api-access-bvl65\") pod \"alertmanager-main-0\" (UID: \"569c6b31-d722-4187-868b-bc4a6f019c44\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:25.062047 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:25.062008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 19:05:25.210283 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:25.210245 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 19:05:25.394004 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:05:25.393971 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569c6b31_d722_4187_868b_bc4a6f019c44.slice/crio-1a67f158e9863b846092a029fd9b1cdb9154f1fd5b166411c364cf6c3be11300 WatchSource:0}: Error finding container 1a67f158e9863b846092a029fd9b1cdb9154f1fd5b166411c364cf6c3be11300: Status 404 returned error can't find the container with id 1a67f158e9863b846092a029fd9b1cdb9154f1fd5b166411c364cf6c3be11300 Jul 29 19:05:25.487157 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:25.487126 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chrfj" event={"ID":"833b3fa9-8636-4eda-a9d4-543a83510835","Type":"ContainerStarted","Data":"f08688b1e50e31fbde4fc5434cb1294ecd7b91d816facef9c8c083e7931d24ad"} Jul 29 19:05:25.488117 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:25.488095 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"569c6b31-d722-4187-868b-bc4a6f019c44","Type":"ContainerStarted","Data":"1a67f158e9863b846092a029fd9b1cdb9154f1fd5b166411c364cf6c3be11300"} Jul 29 19:05:26.494486 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.494449 2575 generic.go:356] "Generic (PLEG): container finished" podID="833b3fa9-8636-4eda-a9d4-543a83510835" containerID="533aad982bcf140f4d70ea643b1d50ba472c5eb4e2a577f684e4db91b81cc31e" exitCode=0 Jul 29 19:05:26.494955 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.494549 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chrfj" event={"ID":"833b3fa9-8636-4eda-a9d4-543a83510835","Type":"ContainerDied","Data":"533aad982bcf140f4d70ea643b1d50ba472c5eb4e2a577f684e4db91b81cc31e"} Jul 29 19:05:26.736540 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.736495 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84447cf48b-z2d69"] Jul 29 19:05:26.740250 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.740222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.742690 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.742659 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 29 19:05:26.742842 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.742691 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zn7tp\"" Jul 29 19:05:26.742842 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.742722 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 29 19:05:26.742997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.742872 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 29 19:05:26.742997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.742982 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 29 19:05:26.743120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.743057 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 29 19:05:26.743194 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.743176 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c7m66brdn3t5l\"" Jul 29 19:05:26.753555 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.753505 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84447cf48b-z2d69"] Jul 29 19:05:26.866155 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.866133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-grpc-tls\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.866260 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.866181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.866260 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.866212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-tls\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.866346 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.866270 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhx5\" (UniqueName: \"kubernetes.io/projected/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-kube-api-access-gkhx5\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.866346 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.866302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.866346 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.866329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.866567 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.866357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-metrics-client-ca\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.868565 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.866811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.971552 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.971501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-grpc-tls\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.971702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.971592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.971702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.971631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-tls\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.971702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.971663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhx5\" (UniqueName: \"kubernetes.io/projected/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-kube-api-access-gkhx5\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.971858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.971698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.971858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.971723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.971858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.971763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-metrics-client-ca\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.971858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.971792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.972581 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.972553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-metrics-client-ca\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.974396 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.974365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.974483 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.974397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.974554 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.974538 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-grpc-tls\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.974606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.974555 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-tls\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.974655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.974628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.974697 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.974671 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:26.979662 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:26.979639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhx5\" (UniqueName: \"kubernetes.io/projected/3bed8f20-6b8a-4a45-82ad-313c6c9ff53e-kube-api-access-gkhx5\") pod \"thanos-querier-84447cf48b-z2d69\" (UID: \"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e\") " pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:27.051385 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.051309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:27.187779 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.187740 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84447cf48b-z2d69"] Jul 29 19:05:27.191365 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:05:27.191333 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bed8f20_6b8a_4a45_82ad_313c6c9ff53e.slice/crio-92403d5a46269c30b8629c6ce40b51013a2fb5dd2a986143b52ff6765fdb5c32 WatchSource:0}: Error finding container 92403d5a46269c30b8629c6ce40b51013a2fb5dd2a986143b52ff6765fdb5c32: Status 404 returned error can't find the container with id 92403d5a46269c30b8629c6ce40b51013a2fb5dd2a986143b52ff6765fdb5c32 Jul 29 19:05:27.229420 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.229365 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" podUID="78cbea3f-8027-4406-aa1f-cb3ba988d9ae" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 19:05:27.499728 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.499693 2575 generic.go:356] "Generic (PLEG): container finished" podID="569c6b31-d722-4187-868b-bc4a6f019c44" containerID="537bdf28f7168e4e39d4560a722ab830975abfd89e71aa23d577e114e01b076b" exitCode=0 Jul 29 19:05:27.500156 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.499784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"569c6b31-d722-4187-868b-bc4a6f019c44","Type":"ContainerDied","Data":"537bdf28f7168e4e39d4560a722ab830975abfd89e71aa23d577e114e01b076b"} Jul 29 19:05:27.501040 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.501017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" event={"ID":"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e","Type":"ContainerStarted","Data":"92403d5a46269c30b8629c6ce40b51013a2fb5dd2a986143b52ff6765fdb5c32"} Jul 29 19:05:27.503356 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.503259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chrfj" event={"ID":"833b3fa9-8636-4eda-a9d4-543a83510835","Type":"ContainerStarted","Data":"94c47fa848eb8c65ab4efc0d9023dd5e813669c028c3a018e751e824376136cc"} Jul 29 19:05:27.503356 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.503306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chrfj" event={"ID":"833b3fa9-8636-4eda-a9d4-543a83510835","Type":"ContainerStarted","Data":"b97d8b71259c3ced63bdd8e9399d7251db627ad990de41d46feb7d2f677cae1a"} Jul 29 19:05:27.539282 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:27.539230 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-chrfj" podStartSLOduration=3.630610872 podStartE2EDuration="4.539215881s" totalImagesPullingTime="845.022664ms" totalInitContainerRuntime="63.582345ms" isStatefulPod=true podCreationTimestamp="2026-07-29 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:27.538796092 +0000 UTC m=+308.555817203" watchObservedRunningTime="2026-07-29 19:05:27.539215881 +0000 UTC m=+308.556236977" Jul 29 19:05:28.429300 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:28.429272 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:05:30.517389 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:30.517317 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" event={"ID":"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e","Type":"ContainerStarted","Data":"bd97ae6af711672602fc732e805ba88d06fc18089983fbbca94761387fc5ce99"} Jul 29 19:05:30.517742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:30.517408 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" event={"ID":"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e","Type":"ContainerStarted","Data":"9a6edf418f2c56d63d3b6206f601777560f20f65663604cab9c98b1512c7b236"} Jul 29 19:05:30.517742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:30.517435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" event={"ID":"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e","Type":"ContainerStarted","Data":"b21aa939429419ebe2d660c442aff136efaf406a6cd4cdff6a12a0e879a67ca5"} Jul 29 19:05:30.543523 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:30.543493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:05:32.528683 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.528645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"569c6b31-d722-4187-868b-bc4a6f019c44","Type":"ContainerStarted","Data":"86720fb3591951da6897a59f1b90a190d6ec9c8f8bcd172b60622258d4fe1b34"} Jul 29 19:05:32.529061 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.528742 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"569c6b31-d722-4187-868b-bc4a6f019c44","Type":"ContainerStarted","Data":"43ee2420dc479bf1e3004e8f03e6c7574e0b7eee3687d3b2e470589f9adf02f8"} Jul 29 19:05:32.529061 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.528787 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"569c6b31-d722-4187-868b-bc4a6f019c44","Type":"ContainerStarted","Data":"6340643b9fca7d321678811b41165552a818bf0d9a27fbbe58bf87d4e9c263e2"} Jul 29 19:05:32.531553 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.531505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" event={"ID":"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e","Type":"ContainerStarted","Data":"da53a9f5e52bcf167c8d2aab11aa5d6c592f94bbd6be19fcd77b4d22da435609"} Jul 29 19:05:32.531654 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.531595 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" event={"ID":"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e","Type":"ContainerStarted","Data":"b112d132353d98f4addcacd9b1769cde0a5975f3b7696b61cb3eaa3e068c763f"} Jul 29 19:05:32.531654 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.531650 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" event={"ID":"3bed8f20-6b8a-4a45-82ad-313c6c9ff53e","Type":"ContainerStarted","Data":"8cb175c32af547fa454ced5130711ca5f83ac246f7601ec54bde694b329d28d7"} Jul 29 19:05:32.531734 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.531719 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:32.561781 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.559131 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" podStartSLOduration=1.991262251 podStartE2EDuration="6.559108336s" totalImagesPullingTime="4.567846085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:32.55495313 +0000 UTC m=+313.571974226" watchObservedRunningTime="2026-07-29 19:05:32.559108336 +0000 UTC m=+313.576129434" Jul 29 19:05:32.567673 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:32.567618 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-758bf44b74-j7ths"] Jul 29 19:05:33.538529 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:33.538488 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"569c6b31-d722-4187-868b-bc4a6f019c44","Type":"ContainerStarted","Data":"afa7d8423cdaf5a5e35319357e124d21a02b684e9b890691a5076d071a4ef926"} Jul 29 19:05:33.538932 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:33.538574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"569c6b31-d722-4187-868b-bc4a6f019c44","Type":"ContainerStarted","Data":"324253174f05d4e66ad83d9b65983a84804533192684325d6b8319db4088f477"} Jul 29 19:05:33.538932 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:33.538608 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"569c6b31-d722-4187-868b-bc4a6f019c44","Type":"ContainerStarted","Data":"a727c8b8da4b60d6a06ddbcd11d42755adcdcfec72209cd25cb8133dd619eeeb"} Jul 29 19:05:33.566274 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:33.566219 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.50826101 podStartE2EDuration="9.566199121s" totalImagesPullingTime="6.039977391s" totalInitContainerRuntime="17.96072ms" isStatefulPod=false podCreationTimestamp="2026-07-29 19:05:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:05:33.564351592 +0000 UTC m=+314.581372689" watchObservedRunningTime="2026-07-29 19:05:33.566199121 +0000 UTC m=+314.583220217" Jul 29 19:05:37.229689 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:37.229646 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" podUID="78cbea3f-8027-4406-aa1f-cb3ba988d9ae" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 19:05:37.230132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:37.229716 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" Jul 29 19:05:37.230247 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:37.230216 2575 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f6a7ee75b460393e0fd5a5ae444b88a9483af4194204377c97767d81a57a2ea8"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 29 19:05:37.230284 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:37.230269 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" podUID="78cbea3f-8027-4406-aa1f-cb3ba988d9ae" containerName="service-proxy" containerID="cri-o://f6a7ee75b460393e0fd5a5ae444b88a9483af4194204377c97767d81a57a2ea8" gracePeriod=30 Jul 29 19:05:37.554446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:37.554352 2575 generic.go:356] "Generic (PLEG): container finished" podID="78cbea3f-8027-4406-aa1f-cb3ba988d9ae" containerID="f6a7ee75b460393e0fd5a5ae444b88a9483af4194204377c97767d81a57a2ea8" exitCode=2 Jul 29 19:05:37.554642 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:37.554432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" event={"ID":"78cbea3f-8027-4406-aa1f-cb3ba988d9ae","Type":"ContainerDied","Data":"f6a7ee75b460393e0fd5a5ae444b88a9483af4194204377c97767d81a57a2ea8"} Jul 29 19:05:37.554642 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:37.554492 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c4dccbbcc-wvfpl" event={"ID":"78cbea3f-8027-4406-aa1f-cb3ba988d9ae","Type":"ContainerStarted","Data":"8774fb98f94fcbca5a070bccabd39f7318ea2a219a9a20816903d68919d19e96"} Jul 29 19:05:38.545256 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:38.545225 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84447cf48b-z2d69" Jul 29 19:05:57.588311 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.588245 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" podUID="935916f1-b02b-4ae4-965a-7a6e508daf22" containerName="registry" containerID="cri-o://71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf" gracePeriod=30 Jul 29 19:05:57.830798 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.830772 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:05:57.929289 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929254 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") pod \"935916f1-b02b-4ae4-965a-7a6e508daf22\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " Jul 29 19:05:57.929504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929324 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv9sh\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-kube-api-access-zv9sh\") pod \"935916f1-b02b-4ae4-965a-7a6e508daf22\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " Jul 29 19:05:57.929504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929350 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-trusted-ca\") pod \"935916f1-b02b-4ae4-965a-7a6e508daf22\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " Jul 29 19:05:57.929653 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929505 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935916f1-b02b-4ae4-965a-7a6e508daf22-ca-trust-extracted\") pod \"935916f1-b02b-4ae4-965a-7a6e508daf22\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " Jul 29 19:05:57.929653 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929628 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-bound-sa-token\") pod \"935916f1-b02b-4ae4-965a-7a6e508daf22\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " Jul 29 19:05:57.929755 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929668 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-certificates\") pod \"935916f1-b02b-4ae4-965a-7a6e508daf22\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " Jul 29 19:05:57.929755 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929704 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-image-registry-private-configuration\") pod \"935916f1-b02b-4ae4-965a-7a6e508daf22\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " Jul 29 19:05:57.929755 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929735 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-installation-pull-secrets\") pod \"935916f1-b02b-4ae4-965a-7a6e508daf22\" (UID: \"935916f1-b02b-4ae4-965a-7a6e508daf22\") " Jul 29 19:05:57.929909 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.929783 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "935916f1-b02b-4ae4-965a-7a6e508daf22" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:05:57.930029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.930007 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-trusted-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:05:57.930638 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.930615 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "935916f1-b02b-4ae4-965a-7a6e508daf22" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:05:57.932054 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.932023 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "935916f1-b02b-4ae4-965a-7a6e508daf22" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:05:57.932163 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.932070 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-kube-api-access-zv9sh" (OuterVolumeSpecName: "kube-api-access-zv9sh") pod "935916f1-b02b-4ae4-965a-7a6e508daf22" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22"). InnerVolumeSpecName "kube-api-access-zv9sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:05:57.932233 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.932153 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "935916f1-b02b-4ae4-965a-7a6e508daf22" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:05:57.932233 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.932171 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "935916f1-b02b-4ae4-965a-7a6e508daf22" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:05:57.932336 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.932310 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "935916f1-b02b-4ae4-965a-7a6e508daf22" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:05:57.938695 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:57.938671 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935916f1-b02b-4ae4-965a-7a6e508daf22-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "935916f1-b02b-4ae4-965a-7a6e508daf22" (UID: "935916f1-b02b-4ae4-965a-7a6e508daf22"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:05:58.030926 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.030892 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv9sh\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-kube-api-access-zv9sh\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:05:58.030926 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.030920 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935916f1-b02b-4ae4-965a-7a6e508daf22-ca-trust-extracted\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:05:58.030926 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.030930 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-bound-sa-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:05:58.031140 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.030939 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-certificates\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:05:58.031140 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.030949 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-image-registry-private-configuration\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:05:58.031140 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.030957 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935916f1-b02b-4ae4-965a-7a6e508daf22-installation-pull-secrets\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:05:58.031140 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.030967 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935916f1-b02b-4ae4-965a-7a6e508daf22-registry-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:05:58.630362 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.630323 2575 generic.go:356] "Generic (PLEG): container finished" podID="935916f1-b02b-4ae4-965a-7a6e508daf22" containerID="71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf" exitCode=0 Jul 29 19:05:58.630881 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.630370 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" Jul 29 19:05:58.630881 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.630375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" event={"ID":"935916f1-b02b-4ae4-965a-7a6e508daf22","Type":"ContainerDied","Data":"71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf"} Jul 29 19:05:58.630881 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.630426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758bf44b74-j7ths" event={"ID":"935916f1-b02b-4ae4-965a-7a6e508daf22","Type":"ContainerDied","Data":"e769908ce1371bea05e69016b1b6515743ab3a41b4353224621a01a0e80487d8"} Jul 29 19:05:58.630881 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.630447 2575 scope.go:117] "RemoveContainer" containerID="71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf" Jul 29 19:05:58.638992 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.638974 2575 scope.go:117] "RemoveContainer" containerID="71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf" Jul 29 19:05:58.639270 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:05:58.639250 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf\": container with ID starting with 71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf not found: ID does not exist" containerID="71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf" Jul 29 19:05:58.639336 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.639284 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf"} err="failed to get container status \"71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf\": rpc error: code = NotFound desc = could not find container \"71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf\": container with ID starting with 71562fb10a96c8220222b4c27bd50a24cfb95a3a5ee48da007e064f09e7ad8bf not found: ID does not exist" Jul 29 19:05:58.650551 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.650509 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-758bf44b74-j7ths"] Jul 29 19:05:58.655725 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:58.655689 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-758bf44b74-j7ths"] Jul 29 19:05:59.547439 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:05:59.547400 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="935916f1-b02b-4ae4-965a-7a6e508daf22" path="/var/lib/kubelet/pods/935916f1-b02b-4ae4-965a-7a6e508daf22/volumes" Jul 29 19:06:33.414276 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:33.414223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:06:33.416570 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:33.416549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11505d8a-ce7a-4382-bdf7-9d77110b4ef0-metrics-certs\") pod \"network-metrics-daemon-ldxw2\" (UID: \"11505d8a-ce7a-4382-bdf7-9d77110b4ef0\") " pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:06:33.547466 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:33.547438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qh2tz\"" Jul 29 19:06:33.554892 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:33.554864 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldxw2" Jul 29 19:06:33.681260 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:33.681184 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldxw2"] Jul 29 19:06:33.687257 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:06:33.684944 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11505d8a_ce7a_4382_bdf7_9d77110b4ef0.slice/crio-02b3c1a29a8ed8c23af2a1084b42ec96f314bbde09c404524d99869c7982cde6 WatchSource:0}: Error finding container 02b3c1a29a8ed8c23af2a1084b42ec96f314bbde09c404524d99869c7982cde6: Status 404 returned error can't find the container with id 02b3c1a29a8ed8c23af2a1084b42ec96f314bbde09c404524d99869c7982cde6 Jul 29 19:06:33.758401 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:33.758371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldxw2" event={"ID":"11505d8a-ce7a-4382-bdf7-9d77110b4ef0","Type":"ContainerStarted","Data":"02b3c1a29a8ed8c23af2a1084b42ec96f314bbde09c404524d99869c7982cde6"} Jul 29 19:06:34.764283 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:34.764245 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldxw2" event={"ID":"11505d8a-ce7a-4382-bdf7-9d77110b4ef0","Type":"ContainerStarted","Data":"67c8069955533b93ca109fbebb4f57243379067f00d4c7a4d8d7630a2a6f2cd4"} Jul 29 19:06:34.764688 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:34.764331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldxw2" event={"ID":"11505d8a-ce7a-4382-bdf7-9d77110b4ef0","Type":"ContainerStarted","Data":"7b422ccae9ef9d6ccdc5d5172352b173ca77752b4daf95ac392638af656ec33f"} Jul 29 19:06:34.780714 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:34.780620 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ldxw2" podStartSLOduration=374.938494237 podStartE2EDuration="6m15.780605113s" totalImagesPullingTime="842.110876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:06:34.778717635 +0000 UTC m=+375.795738744" watchObservedRunningTime="2026-07-29 19:06:34.780605113 +0000 UTC m=+375.797626229" Jul 29 19:06:47.986216 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.986182 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bbd786858-blfw8"] Jul 29 19:06:47.986659 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.986443 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935916f1-b02b-4ae4-965a-7a6e508daf22" containerName="registry" Jul 29 19:06:47.986659 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.986453 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="935916f1-b02b-4ae4-965a-7a6e508daf22" containerName="registry" Jul 29 19:06:47.986659 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.986509 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="935916f1-b02b-4ae4-965a-7a6e508daf22" containerName="registry" Jul 29 19:06:47.989391 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.989374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:47.991773 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.991751 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 19:06:47.991985 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.991964 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 19:06:47.992240 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.992218 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 19:06:47.992578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.992239 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 19:06:47.992578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.992227 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 19:06:47.992578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.992270 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zp9v5\"" Jul 29 19:06:47.997855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:47.997836 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 19:06:48.002107 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.002085 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bbd786858-blfw8"] Jul 29 19:06:48.127093 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.127048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-metrics-client-ca\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.127268 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.127098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64n2f\" (UniqueName: \"kubernetes.io/projected/f5edd272-19b9-440e-a5b1-a2ae119d11af-kube-api-access-64n2f\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.127268 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.127166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.127268 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.127231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-secret-telemeter-client\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.127268 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.127259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.127437 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.127294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-serving-certs-ca-bundle\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.127437 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.127318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-federate-client-tls\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.127437 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.127343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-telemeter-client-tls\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.228410 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.228355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-telemeter-client-tls\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.228633 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.228426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-metrics-client-ca\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.228633 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.228453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64n2f\" (UniqueName: \"kubernetes.io/projected/f5edd272-19b9-440e-a5b1-a2ae119d11af-kube-api-access-64n2f\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.228633 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.228486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.228633 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.228567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-secret-telemeter-client\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.228633 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.228595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.228960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.228635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-serving-certs-ca-bundle\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.228960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.228668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-federate-client-tls\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.229281 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.229257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-metrics-client-ca\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.229464 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.229356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-serving-certs-ca-bundle\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.229934 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.229910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5edd272-19b9-440e-a5b1-a2ae119d11af-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.231170 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.231126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-secret-telemeter-client\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.231170 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.231159 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-telemeter-client-tls\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.231394 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.231375 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-federate-client-tls\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.231446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.231392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5edd272-19b9-440e-a5b1-a2ae119d11af-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.237600 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.237561 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64n2f\" (UniqueName: \"kubernetes.io/projected/f5edd272-19b9-440e-a5b1-a2ae119d11af-kube-api-access-64n2f\") pod \"telemeter-client-5bbd786858-blfw8\" (UID: \"f5edd272-19b9-440e-a5b1-a2ae119d11af\") " pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.299177 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.299150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" Jul 29 19:06:48.434578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.434541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bbd786858-blfw8"] Jul 29 19:06:48.438096 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:06:48.438067 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5edd272_19b9_440e_a5b1_a2ae119d11af.slice/crio-9028de50a87fbd12b28441fae79caf96b88e5046485f90e828a92fbae56f54ec WatchSource:0}: Error finding container 9028de50a87fbd12b28441fae79caf96b88e5046485f90e828a92fbae56f54ec: Status 404 returned error can't find the container with id 9028de50a87fbd12b28441fae79caf96b88e5046485f90e828a92fbae56f54ec Jul 29 19:06:48.818922 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:48.818881 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" event={"ID":"f5edd272-19b9-440e-a5b1-a2ae119d11af","Type":"ContainerStarted","Data":"9028de50a87fbd12b28441fae79caf96b88e5046485f90e828a92fbae56f54ec"} Jul 29 19:06:50.828460 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:50.828421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" event={"ID":"f5edd272-19b9-440e-a5b1-a2ae119d11af","Type":"ContainerStarted","Data":"a87bbd128f34467940b1892804367fbb399537010d393ab67320cf97322aae93"} Jul 29 19:06:51.834315 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:51.834274 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" event={"ID":"f5edd272-19b9-440e-a5b1-a2ae119d11af","Type":"ContainerStarted","Data":"8d61e6d81af01cda7555f55c63a281bc777ad85c8076421eeb5b1392d8ed5041"} Jul 29 19:06:51.834729 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:51.834343 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" event={"ID":"f5edd272-19b9-440e-a5b1-a2ae119d11af","Type":"ContainerStarted","Data":"e8b7b36803e14f6b243161210f1c7b6f98aa14f6be09a1ed38b15938a1452f90"} Jul 29 19:06:51.860929 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:06:51.860847 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bbd786858-blfw8" podStartSLOduration=2.5637241619999998 podStartE2EDuration="4.860826077s" totalImagesPullingTime="2.297101915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:06:51.859093434 +0000 UTC m=+392.876114540" watchObservedRunningTime="2026-07-29 19:06:51.860826077 +0000 UTC m=+392.877847174" Jul 29 19:08:05.962128 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:05.962093 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j"] Jul 29 19:08:05.964818 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:05.964801 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:05.967492 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:05.967469 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 19:08:05.968546 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:05.968509 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 19:08:05.968546 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:05.968539 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dg2cz\"" Jul 29 19:08:05.974073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:05.974051 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j"] Jul 29 19:08:06.136524 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.136493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.136721 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.136563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gt27\" (UniqueName: \"kubernetes.io/projected/fd85d796-579d-4039-b7eb-e875d1478558-kube-api-access-4gt27\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.136721 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.136651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.238034 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.237947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.238034 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.238012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.238247 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.238046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gt27\" (UniqueName: \"kubernetes.io/projected/fd85d796-579d-4039-b7eb-e875d1478558-kube-api-access-4gt27\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.238404 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.238379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.238464 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.238426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.246424 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.246393 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gt27\" (UniqueName: \"kubernetes.io/projected/fd85d796-579d-4039-b7eb-e875d1478558-kube-api-access-4gt27\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.273975 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.273947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:06.400765 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:06.400729 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j"] Jul 29 19:08:06.403798 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:08:06.403774 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd85d796_579d_4039_b7eb_e875d1478558.slice/crio-22f93f83a52b3ca695d49e589e7b8fd7bf93a5be587c0a8ebf5e29ec070176e7 WatchSource:0}: Error finding container 22f93f83a52b3ca695d49e589e7b8fd7bf93a5be587c0a8ebf5e29ec070176e7: Status 404 returned error can't find the container with id 22f93f83a52b3ca695d49e589e7b8fd7bf93a5be587c0a8ebf5e29ec070176e7 Jul 29 19:08:07.106858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:07.106819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" event={"ID":"fd85d796-579d-4039-b7eb-e875d1478558","Type":"ContainerStarted","Data":"22f93f83a52b3ca695d49e589e7b8fd7bf93a5be587c0a8ebf5e29ec070176e7"} Jul 29 19:08:15.140395 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:15.140362 2575 generic.go:356] "Generic (PLEG): container finished" podID="fd85d796-579d-4039-b7eb-e875d1478558" containerID="5edf6240c2c92f70fd663cdbfa7203e0200878727aba84622603f2e492f001a1" exitCode=0 Jul 29 19:08:15.140826 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:15.140434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" event={"ID":"fd85d796-579d-4039-b7eb-e875d1478558","Type":"ContainerDied","Data":"5edf6240c2c92f70fd663cdbfa7203e0200878727aba84622603f2e492f001a1"} Jul 29 19:08:18.157970 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:18.157936 2575 generic.go:356] "Generic (PLEG): container finished" podID="fd85d796-579d-4039-b7eb-e875d1478558" containerID="f1b91999f6bc0460f50c12c14994b2c3315db1f709ba557b50ea62a34bf5d28f" exitCode=0 Jul 29 19:08:18.158359 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:18.158023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" event={"ID":"fd85d796-579d-4039-b7eb-e875d1478558","Type":"ContainerDied","Data":"f1b91999f6bc0460f50c12c14994b2c3315db1f709ba557b50ea62a34bf5d28f"} Jul 29 19:08:26.191610 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:26.191575 2575 generic.go:356] "Generic (PLEG): container finished" podID="fd85d796-579d-4039-b7eb-e875d1478558" containerID="24d33841d6fabfcab48b9a9e70e6ec0dfb218890ffa0b5e888c52ccac396b5a7" exitCode=0 Jul 29 19:08:26.191610 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:26.191614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" event={"ID":"fd85d796-579d-4039-b7eb-e875d1478558","Type":"ContainerDied","Data":"24d33841d6fabfcab48b9a9e70e6ec0dfb218890ffa0b5e888c52ccac396b5a7"} Jul 29 19:08:27.324667 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.324639 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:27.387840 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.387812 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gt27\" (UniqueName: \"kubernetes.io/projected/fd85d796-579d-4039-b7eb-e875d1478558-kube-api-access-4gt27\") pod \"fd85d796-579d-4039-b7eb-e875d1478558\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " Jul 29 19:08:27.388026 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.387879 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-util\") pod \"fd85d796-579d-4039-b7eb-e875d1478558\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " Jul 29 19:08:27.388026 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.387896 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-bundle\") pod \"fd85d796-579d-4039-b7eb-e875d1478558\" (UID: \"fd85d796-579d-4039-b7eb-e875d1478558\") " Jul 29 19:08:27.388602 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.388575 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-bundle" (OuterVolumeSpecName: "bundle") pod "fd85d796-579d-4039-b7eb-e875d1478558" (UID: "fd85d796-579d-4039-b7eb-e875d1478558"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:08:27.390016 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.389987 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd85d796-579d-4039-b7eb-e875d1478558-kube-api-access-4gt27" (OuterVolumeSpecName: "kube-api-access-4gt27") pod "fd85d796-579d-4039-b7eb-e875d1478558" (UID: "fd85d796-579d-4039-b7eb-e875d1478558"). InnerVolumeSpecName "kube-api-access-4gt27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:08:27.392248 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.392229 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-util" (OuterVolumeSpecName: "util") pod "fd85d796-579d-4039-b7eb-e875d1478558" (UID: "fd85d796-579d-4039-b7eb-e875d1478558"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:08:27.489314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.489226 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gt27\" (UniqueName: \"kubernetes.io/projected/fd85d796-579d-4039-b7eb-e875d1478558-kube-api-access-4gt27\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:27.489314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.489265 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-util\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:27.489314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:27.489275 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd85d796-579d-4039-b7eb-e875d1478558-bundle\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:28.200582 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:28.200548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" event={"ID":"fd85d796-579d-4039-b7eb-e875d1478558","Type":"ContainerDied","Data":"22f93f83a52b3ca695d49e589e7b8fd7bf93a5be587c0a8ebf5e29ec070176e7"} Jul 29 19:08:28.200582 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:28.200578 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bm55j" Jul 29 19:08:28.200797 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:28.200599 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f93f83a52b3ca695d49e589e7b8fd7bf93a5be587c0a8ebf5e29ec070176e7" Jul 29 19:08:34.311406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.311368 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v"] Jul 29 19:08:34.311863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.311664 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd85d796-579d-4039-b7eb-e875d1478558" containerName="pull" Jul 29 19:08:34.311863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.311676 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd85d796-579d-4039-b7eb-e875d1478558" containerName="pull" Jul 29 19:08:34.311863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.311684 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd85d796-579d-4039-b7eb-e875d1478558" containerName="extract" Jul 29 19:08:34.311863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.311691 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd85d796-579d-4039-b7eb-e875d1478558" containerName="extract" Jul 29 19:08:34.311863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.311701 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd85d796-579d-4039-b7eb-e875d1478558" containerName="util" Jul 29 19:08:34.311863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.311706 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd85d796-579d-4039-b7eb-e875d1478558" containerName="util" Jul 29 19:08:34.311863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.311746 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd85d796-579d-4039-b7eb-e875d1478558" containerName="extract" Jul 29 19:08:34.315966 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.315949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" Jul 29 19:08:34.319236 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.319213 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 29 19:08:34.319504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.319486 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 29 19:08:34.319804 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.319792 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-n2hlq\"" Jul 29 19:08:34.343316 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.343293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v"] Jul 29 19:08:34.345221 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.345196 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6thg\" (UniqueName: \"kubernetes.io/projected/ef4fb607-b37f-4a0a-983d-3afe302ee702-kube-api-access-n6thg\") pod \"cert-manager-operator-controller-manager-55c444bf86-kxk6v\" (UID: \"ef4fb607-b37f-4a0a-983d-3afe302ee702\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" Jul 29 19:08:34.345288 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.345252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef4fb607-b37f-4a0a-983d-3afe302ee702-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kxk6v\" (UID: \"ef4fb607-b37f-4a0a-983d-3afe302ee702\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" Jul 29 19:08:34.446346 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.446314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6thg\" (UniqueName: \"kubernetes.io/projected/ef4fb607-b37f-4a0a-983d-3afe302ee702-kube-api-access-n6thg\") pod \"cert-manager-operator-controller-manager-55c444bf86-kxk6v\" (UID: \"ef4fb607-b37f-4a0a-983d-3afe302ee702\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" Jul 29 19:08:34.446501 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.446357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef4fb607-b37f-4a0a-983d-3afe302ee702-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kxk6v\" (UID: \"ef4fb607-b37f-4a0a-983d-3afe302ee702\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" Jul 29 19:08:34.446753 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.446736 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef4fb607-b37f-4a0a-983d-3afe302ee702-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kxk6v\" (UID: \"ef4fb607-b37f-4a0a-983d-3afe302ee702\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" Jul 29 19:08:34.457167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.457140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6thg\" (UniqueName: \"kubernetes.io/projected/ef4fb607-b37f-4a0a-983d-3afe302ee702-kube-api-access-n6thg\") pod \"cert-manager-operator-controller-manager-55c444bf86-kxk6v\" (UID: \"ef4fb607-b37f-4a0a-983d-3afe302ee702\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" Jul 29 19:08:34.624848 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.624823 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" Jul 29 19:08:34.761003 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:34.760867 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v"] Jul 29 19:08:34.763845 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:08:34.763817 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4fb607_b37f_4a0a_983d_3afe302ee702.slice/crio-fa166a421909d3002f183c1178dfc37afbca2c1886700e554e2930642429249a WatchSource:0}: Error finding container fa166a421909d3002f183c1178dfc37afbca2c1886700e554e2930642429249a: Status 404 returned error can't find the container with id fa166a421909d3002f183c1178dfc37afbca2c1886700e554e2930642429249a Jul 29 19:08:35.227318 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:35.227286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" event={"ID":"ef4fb607-b37f-4a0a-983d-3afe302ee702","Type":"ContainerStarted","Data":"fa166a421909d3002f183c1178dfc37afbca2c1886700e554e2930642429249a"} Jul 29 19:08:37.237524 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:37.237472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" event={"ID":"ef4fb607-b37f-4a0a-983d-3afe302ee702","Type":"ContainerStarted","Data":"5a6a1c326b436c63f4cbad5bf827be70056d1b8309e309b2d11bbb5567ec2198"} Jul 29 19:08:38.523458 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.523400 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kxk6v" podStartSLOduration=2.849877909 podStartE2EDuration="4.523380636s" totalImagesPullingTime="1.673502727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:08:37.275364107 +0000 UTC m=+498.292385202" watchObservedRunningTime="2026-07-29 19:08:38.523380636 +0000 UTC m=+499.540401731" Jul 29 19:08:38.524319 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.524299 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8"] Jul 29 19:08:38.526641 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.526625 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.530894 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.530873 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dg2cz\"" Jul 29 19:08:38.531181 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.531161 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 19:08:38.531838 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.531821 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 19:08:38.556147 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.556121 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8"] Jul 29 19:08:38.575938 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.575908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.576043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.575949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.576043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.576014 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnt7r\" (UniqueName: \"kubernetes.io/projected/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-kube-api-access-dnt7r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.676930 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.676890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.676930 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.676935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.677087 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.676957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt7r\" (UniqueName: \"kubernetes.io/projected/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-kube-api-access-dnt7r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.677282 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.677265 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.677323 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.677290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.687136 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.687112 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt7r\" (UniqueName: \"kubernetes.io/projected/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-kube-api-access-dnt7r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.836043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.835976 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:38.966941 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:38.966910 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8"] Jul 29 19:08:38.969170 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:08:38.969138 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dab0ee6_f8a6_49bd_94a2_33d16c4a22d3.slice/crio-3a19394499ce47e0226fd47cf41046d3256c70cf6c309940e56d9262f67c7be3 WatchSource:0}: Error finding container 3a19394499ce47e0226fd47cf41046d3256c70cf6c309940e56d9262f67c7be3: Status 404 returned error can't find the container with id 3a19394499ce47e0226fd47cf41046d3256c70cf6c309940e56d9262f67c7be3 Jul 29 19:08:39.246989 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:39.246954 2575 generic.go:356] "Generic (PLEG): container finished" podID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerID="4fae496b833e297da379c15048e3b3c6029639c9f8bd37ba7d1f28c6c26229df" exitCode=0 Jul 29 19:08:39.247159 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:39.247009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" event={"ID":"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3","Type":"ContainerDied","Data":"4fae496b833e297da379c15048e3b3c6029639c9f8bd37ba7d1f28c6c26229df"} Jul 29 19:08:39.247159 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:39.247043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" event={"ID":"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3","Type":"ContainerStarted","Data":"3a19394499ce47e0226fd47cf41046d3256c70cf6c309940e56d9262f67c7be3"} Jul 29 19:08:41.256731 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:41.256699 2575 generic.go:356] "Generic (PLEG): container finished" podID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerID="efcfd47c50c7275268d3cd3043f145c3af23382f17d0dd841960b428b0817bc2" exitCode=0 Jul 29 19:08:41.257135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:41.256777 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" event={"ID":"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3","Type":"ContainerDied","Data":"efcfd47c50c7275268d3cd3043f145c3af23382f17d0dd841960b428b0817bc2"} Jul 29 19:08:42.263249 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:42.263217 2575 generic.go:356] "Generic (PLEG): container finished" podID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerID="5944adb6e2dc682e930e568136f74e4de85b42514e79d0cfd827a7e5e6146939" exitCode=0 Jul 29 19:08:42.263626 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:42.263270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" event={"ID":"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3","Type":"ContainerDied","Data":"5944adb6e2dc682e930e568136f74e4de85b42514e79d0cfd827a7e5e6146939"} Jul 29 19:08:43.338204 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.338177 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89248"] Jul 29 19:08:43.340324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.340302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" Jul 29 19:08:43.343896 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.343873 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-crxzs\"" Jul 29 19:08:43.344989 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.344973 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 19:08:43.345075 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.344974 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 19:08:43.355253 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.355230 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89248"] Jul 29 19:08:43.390084 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.390065 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:43.412397 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.412374 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-bundle\") pod \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " Jul 29 19:08:43.412469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.412409 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-util\") pod \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " Jul 29 19:08:43.412505 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.412481 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnt7r\" (UniqueName: \"kubernetes.io/projected/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-kube-api-access-dnt7r\") pod \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\" (UID: \"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3\") " Jul 29 19:08:43.412597 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.412580 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rg2h\" (UniqueName: \"kubernetes.io/projected/73b85b76-5517-4b73-b658-2ae2dc906039-kube-api-access-6rg2h\") pod \"cert-manager-cainjector-559cdd8c87-89248\" (UID: \"73b85b76-5517-4b73-b658-2ae2dc906039\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" Jul 29 19:08:43.412666 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.412619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73b85b76-5517-4b73-b658-2ae2dc906039-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89248\" (UID: \"73b85b76-5517-4b73-b658-2ae2dc906039\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" Jul 29 19:08:43.412811 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.412780 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-bundle" (OuterVolumeSpecName: "bundle") pod "4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" (UID: "4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:08:43.414566 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.414539 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-kube-api-access-dnt7r" (OuterVolumeSpecName: "kube-api-access-dnt7r") pod "4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" (UID: "4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3"). InnerVolumeSpecName "kube-api-access-dnt7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:08:43.419231 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.419206 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-util" (OuterVolumeSpecName: "util") pod "4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" (UID: "4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:08:43.513468 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.513436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73b85b76-5517-4b73-b658-2ae2dc906039-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89248\" (UID: \"73b85b76-5517-4b73-b658-2ae2dc906039\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" Jul 29 19:08:43.513648 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.513507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rg2h\" (UniqueName: \"kubernetes.io/projected/73b85b76-5517-4b73-b658-2ae2dc906039-kube-api-access-6rg2h\") pod \"cert-manager-cainjector-559cdd8c87-89248\" (UID: \"73b85b76-5517-4b73-b658-2ae2dc906039\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" Jul 29 19:08:43.513648 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.513557 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-bundle\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:43.513648 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.513565 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-util\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:43.513648 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.513575 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnt7r\" (UniqueName: \"kubernetes.io/projected/4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3-kube-api-access-dnt7r\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:43.527253 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.527189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rg2h\" (UniqueName: \"kubernetes.io/projected/73b85b76-5517-4b73-b658-2ae2dc906039-kube-api-access-6rg2h\") pod \"cert-manager-cainjector-559cdd8c87-89248\" (UID: \"73b85b76-5517-4b73-b658-2ae2dc906039\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" Jul 29 19:08:43.528325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.528303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73b85b76-5517-4b73-b658-2ae2dc906039-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89248\" (UID: \"73b85b76-5517-4b73-b658-2ae2dc906039\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" Jul 29 19:08:43.649192 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.649156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" Jul 29 19:08:43.780812 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:43.780737 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89248"] Jul 29 19:08:43.783659 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:08:43.783631 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73b85b76_5517_4b73_b658_2ae2dc906039.slice/crio-23fefca41c43b5c21e3646dd1df90fa1d75d0d2925ac258220c210490ea10d57 WatchSource:0}: Error finding container 23fefca41c43b5c21e3646dd1df90fa1d75d0d2925ac258220c210490ea10d57: Status 404 returned error can't find the container with id 23fefca41c43b5c21e3646dd1df90fa1d75d0d2925ac258220c210490ea10d57 Jul 29 19:08:44.273291 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:44.273254 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" Jul 29 19:08:44.273457 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:44.273254 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fw7jh8" event={"ID":"4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3","Type":"ContainerDied","Data":"3a19394499ce47e0226fd47cf41046d3256c70cf6c309940e56d9262f67c7be3"} Jul 29 19:08:44.273457 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:44.273387 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a19394499ce47e0226fd47cf41046d3256c70cf6c309940e56d9262f67c7be3" Jul 29 19:08:44.274487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:44.274465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" event={"ID":"73b85b76-5517-4b73-b658-2ae2dc906039","Type":"ContainerStarted","Data":"23fefca41c43b5c21e3646dd1df90fa1d75d0d2925ac258220c210490ea10d57"} Jul 29 19:08:48.293223 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.293182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" event={"ID":"73b85b76-5517-4b73-b658-2ae2dc906039","Type":"ContainerStarted","Data":"1fa9b904188e2131ddce6356330ea618829b2ff4f72fb3848155fe8512119788"} Jul 29 19:08:48.319666 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.319617 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89248" podStartSLOduration=1.526740904 podStartE2EDuration="5.319601916s" totalImagesPullingTime="3.792861012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:08:48.317709705 +0000 UTC m=+509.334730802" watchObservedRunningTime="2026-07-29 19:08:48.319601916 +0000 UTC m=+509.336623012" Jul 29 19:08:48.826063 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.826025 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rg2rh"] Jul 29 19:08:48.826341 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.826328 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerName="extract" Jul 29 19:08:48.826380 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.826344 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerName="extract" Jul 29 19:08:48.826380 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.826362 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerName="util" Jul 29 19:08:48.826380 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.826368 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerName="util" Jul 29 19:08:48.826380 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.826377 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerName="pull" Jul 29 19:08:48.826495 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.826383 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerName="pull" Jul 29 19:08:48.826495 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.826431 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dab0ee6-f8a6-49bd-94a2-33d16c4a22d3" containerName="extract" Jul 29 19:08:48.828269 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.828249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rg2rh" Jul 29 19:08:48.830775 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.830751 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-47f59\"" Jul 29 19:08:48.837809 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.837787 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rg2rh"] Jul 29 19:08:48.952989 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.952950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ea5101-bd69-49e2-a5d0-692a433c63ba-bound-sa-token\") pod \"cert-manager-55b4d954d5-rg2rh\" (UID: \"c2ea5101-bd69-49e2-a5d0-692a433c63ba\") " pod="cert-manager/cert-manager-55b4d954d5-rg2rh" Jul 29 19:08:48.953128 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:48.953010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lgq\" (UniqueName: \"kubernetes.io/projected/c2ea5101-bd69-49e2-a5d0-692a433c63ba-kube-api-access-d5lgq\") pod \"cert-manager-55b4d954d5-rg2rh\" (UID: \"c2ea5101-bd69-49e2-a5d0-692a433c63ba\") " pod="cert-manager/cert-manager-55b4d954d5-rg2rh" Jul 29 19:08:49.053947 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:49.053907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ea5101-bd69-49e2-a5d0-692a433c63ba-bound-sa-token\") pod \"cert-manager-55b4d954d5-rg2rh\" (UID: \"c2ea5101-bd69-49e2-a5d0-692a433c63ba\") " pod="cert-manager/cert-manager-55b4d954d5-rg2rh" Jul 29 19:08:49.054130 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:49.053979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lgq\" (UniqueName: \"kubernetes.io/projected/c2ea5101-bd69-49e2-a5d0-692a433c63ba-kube-api-access-d5lgq\") pod \"cert-manager-55b4d954d5-rg2rh\" (UID: \"c2ea5101-bd69-49e2-a5d0-692a433c63ba\") " pod="cert-manager/cert-manager-55b4d954d5-rg2rh" Jul 29 19:08:49.062296 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:49.062272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lgq\" (UniqueName: \"kubernetes.io/projected/c2ea5101-bd69-49e2-a5d0-692a433c63ba-kube-api-access-d5lgq\") pod \"cert-manager-55b4d954d5-rg2rh\" (UID: \"c2ea5101-bd69-49e2-a5d0-692a433c63ba\") " pod="cert-manager/cert-manager-55b4d954d5-rg2rh" Jul 29 19:08:49.063561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:49.063540 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ea5101-bd69-49e2-a5d0-692a433c63ba-bound-sa-token\") pod \"cert-manager-55b4d954d5-rg2rh\" (UID: \"c2ea5101-bd69-49e2-a5d0-692a433c63ba\") " pod="cert-manager/cert-manager-55b4d954d5-rg2rh" Jul 29 19:08:49.138204 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:49.138179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rg2rh" Jul 29 19:08:49.273005 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:49.272971 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rg2rh"] Jul 29 19:08:49.276773 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:08:49.276744 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ea5101_bd69_49e2_a5d0_692a433c63ba.slice/crio-bcff52c9bd08a80a7b8d3d64264a6b20ee0a155c4b138ce4715765567d8b9c4d WatchSource:0}: Error finding container bcff52c9bd08a80a7b8d3d64264a6b20ee0a155c4b138ce4715765567d8b9c4d: Status 404 returned error can't find the container with id bcff52c9bd08a80a7b8d3d64264a6b20ee0a155c4b138ce4715765567d8b9c4d Jul 29 19:08:49.298308 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:49.298283 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rg2rh" event={"ID":"c2ea5101-bd69-49e2-a5d0-692a433c63ba","Type":"ContainerStarted","Data":"bcff52c9bd08a80a7b8d3d64264a6b20ee0a155c4b138ce4715765567d8b9c4d"} Jul 29 19:08:50.304728 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:50.304689 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rg2rh" event={"ID":"c2ea5101-bd69-49e2-a5d0-692a433c63ba","Type":"ContainerStarted","Data":"9e7fceca1faf92411e8ac85596308612fc7c530f5a4b77b081f9e957fd43667b"} Jul 29 19:08:50.328428 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:50.328377 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rg2rh" podStartSLOduration=2.328361492 podStartE2EDuration="2.328361492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:08:50.326860812 +0000 UTC m=+511.343881908" watchObservedRunningTime="2026-07-29 19:08:50.328361492 +0000 UTC m=+511.345382588" Jul 29 19:08:53.703049 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.703012 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q"] Jul 29 19:08:53.705405 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.705388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.708577 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.708552 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 19:08:53.708843 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.708579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dg2cz\"" Jul 29 19:08:53.714332 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.714282 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 19:08:53.717312 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.717291 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q"] Jul 29 19:08:53.791573 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.791540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.791706 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.791584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.791706 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.791642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qs8c\" (UniqueName: \"kubernetes.io/projected/c20c3668-3f03-4d44-866d-7d6351ece5bf-kube-api-access-5qs8c\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.892047 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.892012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qs8c\" (UniqueName: \"kubernetes.io/projected/c20c3668-3f03-4d44-866d-7d6351ece5bf-kube-api-access-5qs8c\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.892192 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.892070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.892192 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.892096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.892415 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.892400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.892457 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.892439 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:53.904014 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:53.903990 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qs8c\" (UniqueName: \"kubernetes.io/projected/c20c3668-3f03-4d44-866d-7d6351ece5bf-kube-api-access-5qs8c\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:54.018374 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:54.018298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:54.156393 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:54.156363 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q"] Jul 29 19:08:54.159101 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:08:54.159065 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20c3668_3f03_4d44_866d_7d6351ece5bf.slice/crio-4ed4b784385f4a9b4f11527dc399d03304c8c1556a4abca06cb8059045412679 WatchSource:0}: Error finding container 4ed4b784385f4a9b4f11527dc399d03304c8c1556a4abca06cb8059045412679: Status 404 returned error can't find the container with id 4ed4b784385f4a9b4f11527dc399d03304c8c1556a4abca06cb8059045412679 Jul 29 19:08:54.322057 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:54.321989 2575 generic.go:356] "Generic (PLEG): container finished" podID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerID="6678dee10107ba52190d253e537e1a70489a8bc4e1308abd8c08e9f498fca1a9" exitCode=0 Jul 29 19:08:54.322177 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:54.322080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" event={"ID":"c20c3668-3f03-4d44-866d-7d6351ece5bf","Type":"ContainerDied","Data":"6678dee10107ba52190d253e537e1a70489a8bc4e1308abd8c08e9f498fca1a9"} Jul 29 19:08:54.322177 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:54.322137 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" event={"ID":"c20c3668-3f03-4d44-866d-7d6351ece5bf","Type":"ContainerStarted","Data":"4ed4b784385f4a9b4f11527dc399d03304c8c1556a4abca06cb8059045412679"} Jul 29 19:08:55.328640 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:55.328550 2575 generic.go:356] "Generic (PLEG): container finished" podID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerID="ccd6b826280fd47f01fa9602fed4dd4d1743a6f4b3d9845a4a5c5e9a64549ece" exitCode=0 Jul 29 19:08:55.328640 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:55.328629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" event={"ID":"c20c3668-3f03-4d44-866d-7d6351ece5bf","Type":"ContainerDied","Data":"ccd6b826280fd47f01fa9602fed4dd4d1743a6f4b3d9845a4a5c5e9a64549ece"} Jul 29 19:08:56.334960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:56.334927 2575 generic.go:356] "Generic (PLEG): container finished" podID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerID="01b50ccdaf20b72354891450b8539eb3885f9327ed4b8ab7b8218b025ee9a177" exitCode=0 Jul 29 19:08:56.335305 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:56.335005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" event={"ID":"c20c3668-3f03-4d44-866d-7d6351ece5bf","Type":"ContainerDied","Data":"01b50ccdaf20b72354891450b8539eb3885f9327ed4b8ab7b8218b025ee9a177"} Jul 29 19:08:57.492573 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.492529 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:57.625407 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.625358 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-util\") pod \"c20c3668-3f03-4d44-866d-7d6351ece5bf\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " Jul 29 19:08:57.625579 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.625423 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-bundle\") pod \"c20c3668-3f03-4d44-866d-7d6351ece5bf\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " Jul 29 19:08:57.625579 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.625441 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qs8c\" (UniqueName: \"kubernetes.io/projected/c20c3668-3f03-4d44-866d-7d6351ece5bf-kube-api-access-5qs8c\") pod \"c20c3668-3f03-4d44-866d-7d6351ece5bf\" (UID: \"c20c3668-3f03-4d44-866d-7d6351ece5bf\") " Jul 29 19:08:57.626299 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.626263 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-bundle" (OuterVolumeSpecName: "bundle") pod "c20c3668-3f03-4d44-866d-7d6351ece5bf" (UID: "c20c3668-3f03-4d44-866d-7d6351ece5bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:08:57.627493 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.627472 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20c3668-3f03-4d44-866d-7d6351ece5bf-kube-api-access-5qs8c" (OuterVolumeSpecName: "kube-api-access-5qs8c") pod "c20c3668-3f03-4d44-866d-7d6351ece5bf" (UID: "c20c3668-3f03-4d44-866d-7d6351ece5bf"). InnerVolumeSpecName "kube-api-access-5qs8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:08:57.631594 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.631549 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-util" (OuterVolumeSpecName: "util") pod "c20c3668-3f03-4d44-866d-7d6351ece5bf" (UID: "c20c3668-3f03-4d44-866d-7d6351ece5bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:08:57.726481 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.726440 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-util\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:57.726481 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.726467 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20c3668-3f03-4d44-866d-7d6351ece5bf-bundle\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:57.726481 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:57.726477 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qs8c\" (UniqueName: \"kubernetes.io/projected/c20c3668-3f03-4d44-866d-7d6351ece5bf-kube-api-access-5qs8c\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:08:58.344589 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:58.344493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" event={"ID":"c20c3668-3f03-4d44-866d-7d6351ece5bf","Type":"ContainerDied","Data":"4ed4b784385f4a9b4f11527dc399d03304c8c1556a4abca06cb8059045412679"} Jul 29 19:08:58.344589 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:58.344508 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl4m2q" Jul 29 19:08:58.344589 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:08:58.344554 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed4b784385f4a9b4f11527dc399d03304c8c1556a4abca06cb8059045412679" Jul 29 19:09:07.415858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.415821 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9"] Jul 29 19:09:07.416304 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.416115 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerName="pull" Jul 29 19:09:07.416304 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.416125 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerName="pull" Jul 29 19:09:07.416304 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.416138 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerName="util" Jul 29 19:09:07.416304 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.416144 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerName="util" Jul 29 19:09:07.416304 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.416151 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerName="extract" Jul 29 19:09:07.416304 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.416157 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerName="extract" Jul 29 19:09:07.416304 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.416210 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c20c3668-3f03-4d44-866d-7d6351ece5bf" containerName="extract" Jul 29 19:09:07.418642 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.418626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.426969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.426950 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 19:09:07.427571 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.427553 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 19:09:07.427712 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.427695 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b4wmv\"" Jul 29 19:09:07.428776 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.428763 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 19:09:07.430123 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.430110 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 19:09:07.478646 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.478604 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9"] Jul 29 19:09:07.505452 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.505420 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20f0196c-5ad4-450c-858a-89aa31d08a61-manifests\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.505452 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.505450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20f0196c-5ad4-450c-858a-89aa31d08a61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.505647 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.505477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20f0196c-5ad4-450c-858a-89aa31d08a61-webhook-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.505647 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.505585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmgg\" (UniqueName: \"kubernetes.io/projected/20f0196c-5ad4-450c-858a-89aa31d08a61-kube-api-access-chmgg\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.505647 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.505624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20f0196c-5ad4-450c-858a-89aa31d08a61-tmp\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.606410 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.606378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20f0196c-5ad4-450c-858a-89aa31d08a61-webhook-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.606613 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.606431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmgg\" (UniqueName: \"kubernetes.io/projected/20f0196c-5ad4-450c-858a-89aa31d08a61-kube-api-access-chmgg\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.606613 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.606466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20f0196c-5ad4-450c-858a-89aa31d08a61-tmp\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.606613 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.606504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20f0196c-5ad4-450c-858a-89aa31d08a61-manifests\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.606613 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.606557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20f0196c-5ad4-450c-858a-89aa31d08a61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.606953 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.606931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20f0196c-5ad4-450c-858a-89aa31d08a61-manifests\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.608694 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.608673 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20f0196c-5ad4-450c-858a-89aa31d08a61-tmp\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.608959 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.608943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20f0196c-5ad4-450c-858a-89aa31d08a61-webhook-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.609001 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.608979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20f0196c-5ad4-450c-858a-89aa31d08a61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.632716 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.632690 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmgg\" (UniqueName: \"kubernetes.io/projected/20f0196c-5ad4-450c-858a-89aa31d08a61-kube-api-access-chmgg\") pod \"opendatahub-operator-controller-manager-b7c5ccf8f-47hp9\" (UID: \"20f0196c-5ad4-450c-858a-89aa31d08a61\") " pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.728431 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.728351 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:07.892666 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:07.892631 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9"] Jul 29 19:09:07.894496 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:09:07.894465 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f0196c_5ad4_450c_858a_89aa31d08a61.slice/crio-d8d45782168d6b89f8881e40fb690d629f145faa2c0201f75b78acdf8279c58c WatchSource:0}: Error finding container d8d45782168d6b89f8881e40fb690d629f145faa2c0201f75b78acdf8279c58c: Status 404 returned error can't find the container with id d8d45782168d6b89f8881e40fb690d629f145faa2c0201f75b78acdf8279c58c Jul 29 19:09:08.386438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:08.386403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" event={"ID":"20f0196c-5ad4-450c-858a-89aa31d08a61","Type":"ContainerStarted","Data":"d8d45782168d6b89f8881e40fb690d629f145faa2c0201f75b78acdf8279c58c"} Jul 29 19:09:12.408314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:12.408276 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" event={"ID":"20f0196c-5ad4-450c-858a-89aa31d08a61","Type":"ContainerStarted","Data":"f4a0d522a759979152115ce87e5089e30aa596bf50b31bd00624691623d8f887"} Jul 29 19:09:14.418253 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:14.418219 2575 generic.go:356] "Generic (PLEG): container finished" podID="20f0196c-5ad4-450c-858a-89aa31d08a61" containerID="f4a0d522a759979152115ce87e5089e30aa596bf50b31bd00624691623d8f887" exitCode=0 Jul 29 19:09:14.418253 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:14.418259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" event={"ID":"20f0196c-5ad4-450c-858a-89aa31d08a61","Type":"ContainerDied","Data":"f4a0d522a759979152115ce87e5089e30aa596bf50b31bd00624691623d8f887"} Jul 29 19:09:15.424331 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:15.424299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" event={"ID":"20f0196c-5ad4-450c-858a-89aa31d08a61","Type":"ContainerStarted","Data":"48453270e066b99aabc3f97de72d298046b9046c179d37e8e7cea781a2df00e2"} Jul 29 19:09:15.424756 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:15.424505 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:15.448052 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:15.448004 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" podStartSLOduration=2.498387356 podStartE2EDuration="8.447990061s" totalImagesPullingTime="4.896996583s" totalInitContainerRuntime="1.052606122s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:15.446686477 +0000 UTC m=+536.463707571" watchObservedRunningTime="2026-07-29 19:09:15.447990061 +0000 UTC m=+536.465011219" Jul 29 19:09:26.431698 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:26.431665 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b7c5ccf8f-47hp9" Jul 29 19:09:29.007688 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.007652 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts"] Jul 29 19:09:29.019142 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.019116 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.027047 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.027026 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 19:09:29.041301 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.041269 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts"] Jul 29 19:09:29.041800 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.041748 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-85knt\"" Jul 29 19:09:29.041914 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.041780 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 19:09:29.041914 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.041859 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 19:09:29.042149 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.042132 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 19:09:29.086440 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.086413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89bcf019-aa18-43e1-a8ad-81fe99025797-tls-certs\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.086586 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.086452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89bcf019-aa18-43e1-a8ad-81fe99025797-tmp\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.086586 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.086478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdmz\" (UniqueName: \"kubernetes.io/projected/89bcf019-aa18-43e1-a8ad-81fe99025797-kube-api-access-zpdmz\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.187275 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.187240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89bcf019-aa18-43e1-a8ad-81fe99025797-tls-certs\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.187465 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.187297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89bcf019-aa18-43e1-a8ad-81fe99025797-tmp\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.187465 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.187330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdmz\" (UniqueName: \"kubernetes.io/projected/89bcf019-aa18-43e1-a8ad-81fe99025797-kube-api-access-zpdmz\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.189525 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.189482 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89bcf019-aa18-43e1-a8ad-81fe99025797-tmp\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.189740 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.189719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89bcf019-aa18-43e1-a8ad-81fe99025797-tls-certs\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.205669 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.205642 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdmz\" (UniqueName: \"kubernetes.io/projected/89bcf019-aa18-43e1-a8ad-81fe99025797-kube-api-access-zpdmz\") pod \"kube-auth-proxy-5f5845fd56-ht8ts\" (UID: \"89bcf019-aa18-43e1-a8ad-81fe99025797\") " pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.329211 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.329125 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" Jul 29 19:09:29.518795 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.518759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts"] Jul 29 19:09:29.521760 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:09:29.521729 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bcf019_aa18_43e1_a8ad_81fe99025797.slice/crio-47cd6f2f96fed9828b8f79e62a1fb0e4afd83aae403c7b750c88eb8fa4cc75f5 WatchSource:0}: Error finding container 47cd6f2f96fed9828b8f79e62a1fb0e4afd83aae403c7b750c88eb8fa4cc75f5: Status 404 returned error can't find the container with id 47cd6f2f96fed9828b8f79e62a1fb0e4afd83aae403c7b750c88eb8fa4cc75f5 Jul 29 19:09:29.650636 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.650602 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vz6ht"] Jul 29 19:09:29.655144 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.655128 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:29.662981 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.662962 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6kthb\"" Jul 29 19:09:29.663268 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.663254 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 19:09:29.689890 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.689865 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vz6ht"] Jul 29 19:09:29.691324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.691302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-cert\") pod \"odh-model-controller-7998c67dc4-vz6ht\" (UID: \"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:29.691423 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.691336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzqr\" (UniqueName: \"kubernetes.io/projected/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-kube-api-access-tvzqr\") pod \"odh-model-controller-7998c67dc4-vz6ht\" (UID: \"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:29.792710 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.792676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-cert\") pod \"odh-model-controller-7998c67dc4-vz6ht\" (UID: \"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:29.792710 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.792713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzqr\" (UniqueName: \"kubernetes.io/projected/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-kube-api-access-tvzqr\") pod \"odh-model-controller-7998c67dc4-vz6ht\" (UID: \"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:29.792892 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:29.792828 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 19:09:29.792927 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:29.792891 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-cert podName:cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8 nodeName:}" failed. No retries permitted until 2026-07-29 19:09:30.292872723 +0000 UTC m=+551.309893798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-cert") pod "odh-model-controller-7998c67dc4-vz6ht" (UID: "cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8") : secret "odh-model-controller-webhook-cert" not found Jul 29 19:09:29.806526 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:29.806486 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzqr\" (UniqueName: \"kubernetes.io/projected/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-kube-api-access-tvzqr\") pod \"odh-model-controller-7998c67dc4-vz6ht\" (UID: \"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:30.296905 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.296870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-cert\") pod \"odh-model-controller-7998c67dc4-vz6ht\" (UID: \"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:30.299505 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.299480 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8-cert\") pod \"odh-model-controller-7998c67dc4-vz6ht\" (UID: \"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8\") " pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:30.490016 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.489974 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" event={"ID":"89bcf019-aa18-43e1-a8ad-81fe99025797","Type":"ContainerStarted","Data":"47cd6f2f96fed9828b8f79e62a1fb0e4afd83aae403c7b750c88eb8fa4cc75f5"} Jul 29 19:09:30.564673 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.564581 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:30.725702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.725669 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vz6ht"] Jul 29 19:09:30.729059 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:09:30.729017 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf41e7f5_aca7_44cd_a4f6_66c1e08fe1c8.slice/crio-0beb33d87e7290f724bea8545ee824cd511e64bcba5d115fbf8247fc956ccb3a WatchSource:0}: Error finding container 0beb33d87e7290f724bea8545ee824cd511e64bcba5d115fbf8247fc956ccb3a: Status 404 returned error can't find the container with id 0beb33d87e7290f724bea8545ee824cd511e64bcba5d115fbf8247fc956ccb3a Jul 29 19:09:30.921421 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.921387 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-czrkh"] Jul 29 19:09:30.925742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.925725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:30.932183 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.932160 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 19:09:30.932316 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.932255 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mt9fx\"" Jul 29 19:09:30.974718 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:30.974685 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-czrkh"] Jul 29 19:09:31.003906 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.003870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870f2696-fc72-4ad2-93b8-756d37203175-tls-certs\") pod \"model-serving-api-54d89bd5fc-czrkh\" (UID: \"870f2696-fc72-4ad2-93b8-756d37203175\") " pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:31.004071 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.003918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wp6f\" (UniqueName: \"kubernetes.io/projected/870f2696-fc72-4ad2-93b8-756d37203175-kube-api-access-5wp6f\") pod \"model-serving-api-54d89bd5fc-czrkh\" (UID: \"870f2696-fc72-4ad2-93b8-756d37203175\") " pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:31.105337 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.105300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wp6f\" (UniqueName: \"kubernetes.io/projected/870f2696-fc72-4ad2-93b8-756d37203175-kube-api-access-5wp6f\") pod \"model-serving-api-54d89bd5fc-czrkh\" (UID: \"870f2696-fc72-4ad2-93b8-756d37203175\") " pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:31.105567 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.105436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870f2696-fc72-4ad2-93b8-756d37203175-tls-certs\") pod \"model-serving-api-54d89bd5fc-czrkh\" (UID: \"870f2696-fc72-4ad2-93b8-756d37203175\") " pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:31.105646 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:31.105568 2575 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 29 19:09:31.105646 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:31.105632 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f2696-fc72-4ad2-93b8-756d37203175-tls-certs podName:870f2696-fc72-4ad2-93b8-756d37203175 nodeName:}" failed. No retries permitted until 2026-07-29 19:09:31.605611882 +0000 UTC m=+552.622632975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/870f2696-fc72-4ad2-93b8-756d37203175-tls-certs") pod "model-serving-api-54d89bd5fc-czrkh" (UID: "870f2696-fc72-4ad2-93b8-756d37203175") : secret "model-serving-api-tls" not found Jul 29 19:09:31.120703 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.120672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wp6f\" (UniqueName: \"kubernetes.io/projected/870f2696-fc72-4ad2-93b8-756d37203175-kube-api-access-5wp6f\") pod \"model-serving-api-54d89bd5fc-czrkh\" (UID: \"870f2696-fc72-4ad2-93b8-756d37203175\") " pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:31.496631 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.496592 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" event={"ID":"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8","Type":"ContainerStarted","Data":"0beb33d87e7290f724bea8545ee824cd511e64bcba5d115fbf8247fc956ccb3a"} Jul 29 19:09:31.609789 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.609749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870f2696-fc72-4ad2-93b8-756d37203175-tls-certs\") pod \"model-serving-api-54d89bd5fc-czrkh\" (UID: \"870f2696-fc72-4ad2-93b8-756d37203175\") " pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:31.612845 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.612806 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870f2696-fc72-4ad2-93b8-756d37203175-tls-certs\") pod \"model-serving-api-54d89bd5fc-czrkh\" (UID: \"870f2696-fc72-4ad2-93b8-756d37203175\") " pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:31.631139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.630806 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h"] Jul 29 19:09:31.634841 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.634247 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.640487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.640466 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 19:09:31.640827 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.640805 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qfzmc\"" Jul 29 19:09:31.641040 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.641022 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 19:09:31.641225 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.641209 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 19:09:31.642030 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.642005 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 19:09:31.667890 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.667799 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h"] Jul 29 19:09:31.710956 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.710924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.711139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.710967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.711139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.710995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27mt\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-kube-api-access-n27mt\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.711139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.711019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.711139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.711058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.711139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.711097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c214f1-9315-4b57-9e3e-6082bad89146-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.711368 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.711176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.812132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.812033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.812132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.812094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.812132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.812124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.812414 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.812150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27mt\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-kube-api-access-n27mt\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.812414 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.812177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.812414 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.812215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.812414 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.812251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c214f1-9315-4b57-9e3e-6082bad89146-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.813236 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.813187 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.815086 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.815061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c214f1-9315-4b57-9e3e-6082bad89146-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.815193 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.815152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.815441 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.815418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.815625 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.815585 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.821687 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.821662 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27mt\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-kube-api-access-n27mt\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.822766 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.822725 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9xj6h\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:31.835738 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.835713 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:31.947101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:31.946956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:32.005235 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:32.005197 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-czrkh"] Jul 29 19:09:33.303104 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:09:33.303066 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870f2696_fc72_4ad2_93b8_756d37203175.slice/crio-1e6e3f6fb7f371035c98e027e307f96acb009a54ad0c860838cebc471ae874e8 WatchSource:0}: Error finding container 1e6e3f6fb7f371035c98e027e307f96acb009a54ad0c860838cebc471ae874e8: Status 404 returned error can't find the container with id 1e6e3f6fb7f371035c98e027e307f96acb009a54ad0c860838cebc471ae874e8 Jul 29 19:09:33.507180 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:33.507140 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" event={"ID":"870f2696-fc72-4ad2-93b8-756d37203175","Type":"ContainerStarted","Data":"1e6e3f6fb7f371035c98e027e307f96acb009a54ad0c860838cebc471ae874e8"} Jul 29 19:09:34.021702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:34.021675 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h"] Jul 29 19:09:34.023646 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:09:34.023616 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c214f1_9315_4b57_9e3e_6082bad89146.slice/crio-e85231008ec19e6e3f83a38140d8ea49db61621b8ed9f359d33cb0fbf58e3fa2 WatchSource:0}: Error finding container e85231008ec19e6e3f83a38140d8ea49db61621b8ed9f359d33cb0fbf58e3fa2: Status 404 returned error can't find the container with id e85231008ec19e6e3f83a38140d8ea49db61621b8ed9f359d33cb0fbf58e3fa2 Jul 29 19:09:34.514953 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:34.514914 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" event={"ID":"89bcf019-aa18-43e1-a8ad-81fe99025797","Type":"ContainerStarted","Data":"f2047ebb8879ff70d52eb7d11a9e96467d9165b6833aaa413c6184c1547d6a31"} Jul 29 19:09:34.516476 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:34.516449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" event={"ID":"cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8","Type":"ContainerStarted","Data":"bdfdd5121096dbaafef7fb39906fe9b304d15cae948bac4025073263f9955a09"} Jul 29 19:09:34.516638 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:34.516553 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:34.517671 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:34.517641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" event={"ID":"37c214f1-9315-4b57-9e3e-6082bad89146","Type":"ContainerStarted","Data":"e85231008ec19e6e3f83a38140d8ea49db61621b8ed9f359d33cb0fbf58e3fa2"} Jul 29 19:09:34.534166 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:34.534116 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f5845fd56-ht8ts" podStartSLOduration=2.172559022 podStartE2EDuration="6.534101179s" totalImagesPullingTime="4.361542157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:34.53220991 +0000 UTC m=+555.549231006" watchObservedRunningTime="2026-07-29 19:09:34.534101179 +0000 UTC m=+555.551122274" Jul 29 19:09:34.552450 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:34.552388 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" podStartSLOduration=2.392598697 podStartE2EDuration="5.552365842s" totalImagesPullingTime="3.159767145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:34.549991463 +0000 UTC m=+555.567012557" watchObservedRunningTime="2026-07-29 19:09:34.552365842 +0000 UTC m=+555.569386939" Jul 29 19:09:36.752460 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:36.752423 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:09:36.752832 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:36.752503 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:09:37.536249 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:37.536210 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" event={"ID":"870f2696-fc72-4ad2-93b8-756d37203175","Type":"ContainerStarted","Data":"12cc2e5fb4ac85a1405c10dde69722c02fe3a443ea821508960a02af6801c7d1"} Jul 29 19:09:37.536585 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:37.536561 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:09:37.538304 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:37.538280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" event={"ID":"37c214f1-9315-4b57-9e3e-6082bad89146","Type":"ContainerStarted","Data":"6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3"} Jul 29 19:09:37.538496 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:37.538480 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:37.540186 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:37.540151 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9xj6h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 19:09:37.540294 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:37.540215 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" podUID="37c214f1-9315-4b57-9e3e-6082bad89146" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:09:37.566323 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:37.566279 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" podStartSLOduration=4.174621013 podStartE2EDuration="7.566265013s" totalImagesPullingTime="3.391644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:37.563998396 +0000 UTC m=+558.581019518" watchObservedRunningTime="2026-07-29 19:09:37.566265013 +0000 UTC m=+558.583286109" Jul 29 19:09:37.591013 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:37.590962 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" podStartSLOduration=3.864482117 podStartE2EDuration="6.590947909s" totalImagesPullingTime="2.726465792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:37.58975182 +0000 UTC m=+558.606772920" watchObservedRunningTime="2026-07-29 19:09:37.590947909 +0000 UTC m=+558.607969005" Jul 29 19:09:38.313478 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.313437 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9"] Jul 29 19:09:38.318343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.318314 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.320928 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.320907 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7fqdj\"" Jul 29 19:09:38.330636 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.330614 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9"] Jul 29 19:09:38.371792 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.371756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.371792 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.371795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.371991 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.371821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.371991 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.371855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qz7x\" (UniqueName: \"kubernetes.io/projected/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-kube-api-access-4qz7x\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.371991 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.371888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.371991 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.371905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.371991 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.371931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.371991 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.371956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.372193 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.372021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qz7x\" (UniqueName: \"kubernetes.io/projected/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-kube-api-access-4qz7x\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473481 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473923 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473923 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473868 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.473923 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473893 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.474071 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.473938 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.474418 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.474391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.475781 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.475764 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.476171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.476149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.485037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.485018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.495981 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.495960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qz7x\" (UniqueName: \"kubernetes.io/projected/9eba4d7d-63f4-49a4-b90b-02c7888fc0c1-kube-api-access-4qz7x\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9\" (UID: \"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.543510 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.543483 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:38.628937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.628906 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:38.778048 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:38.778023 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9"] Jul 29 19:09:38.780015 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:09:38.779988 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eba4d7d_63f4_49a4_b90b_02c7888fc0c1.slice/crio-f44f44b5c2e356711a47d145c7a803e8361edac18167765c29b5740413ce15cb WatchSource:0}: Error finding container f44f44b5c2e356711a47d145c7a803e8361edac18167765c29b5740413ce15cb: Status 404 returned error can't find the container with id f44f44b5c2e356711a47d145c7a803e8361edac18167765c29b5740413ce15cb Jul 29 19:09:39.549249 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:39.549211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" event={"ID":"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1","Type":"ContainerStarted","Data":"f44f44b5c2e356711a47d145c7a803e8361edac18167765c29b5740413ce15cb"} Jul 29 19:09:40.518407 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.518371 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5"] Jul 29 19:09:40.521708 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.521687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:40.529818 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.529797 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 29 19:09:40.530003 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.529991 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jsbx4\"" Jul 29 19:09:40.584219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.584177 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t"] Jul 29 19:09:40.588111 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.588092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:40.592363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.592342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27c6a3fc-aff8-4c5f-a394-959078adbce1-cert\") pod \"kserve-controller-manager-d6f4b65b8-kc6j5\" (UID: \"27c6a3fc-aff8-4c5f-a394-959078adbce1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:40.592455 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.592383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kb7\" (UniqueName: \"kubernetes.io/projected/27c6a3fc-aff8-4c5f-a394-959078adbce1-kube-api-access-68kb7\") pod \"kserve-controller-manager-d6f4b65b8-kc6j5\" (UID: \"27c6a3fc-aff8-4c5f-a394-959078adbce1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:40.604334 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.604306 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 29 19:09:40.604723 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.604708 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-f2tkg\"" Jul 29 19:09:40.693010 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.692977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5tn\" (UniqueName: \"kubernetes.io/projected/e537131c-098c-4ecc-a4f0-cf0b25d71930-kube-api-access-bd5tn\") pod \"llmisvc-controller-manager-777c6bd7d5-v5h7t\" (UID: \"e537131c-098c-4ecc-a4f0-cf0b25d71930\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:40.693161 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.693033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27c6a3fc-aff8-4c5f-a394-959078adbce1-cert\") pod \"kserve-controller-manager-d6f4b65b8-kc6j5\" (UID: \"27c6a3fc-aff8-4c5f-a394-959078adbce1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:40.693161 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.693078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e537131c-098c-4ecc-a4f0-cf0b25d71930-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v5h7t\" (UID: \"e537131c-098c-4ecc-a4f0-cf0b25d71930\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:40.693231 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:40.693165 2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 29 19:09:40.693231 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.693170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kb7\" (UniqueName: \"kubernetes.io/projected/27c6a3fc-aff8-4c5f-a394-959078adbce1-kube-api-access-68kb7\") pod \"kserve-controller-manager-d6f4b65b8-kc6j5\" (UID: \"27c6a3fc-aff8-4c5f-a394-959078adbce1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:40.693231 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:40.693209 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c6a3fc-aff8-4c5f-a394-959078adbce1-cert podName:27c6a3fc-aff8-4c5f-a394-959078adbce1 nodeName:}" failed. No retries permitted until 2026-07-29 19:09:41.193193387 +0000 UTC m=+562.210214461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/27c6a3fc-aff8-4c5f-a394-959078adbce1-cert") pod "kserve-controller-manager-d6f4b65b8-kc6j5" (UID: "27c6a3fc-aff8-4c5f-a394-959078adbce1") : secret "kserve-webhook-server-cert" not found Jul 29 19:09:40.739436 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.739403 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5"] Jul 29 19:09:40.759801 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.759765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kb7\" (UniqueName: \"kubernetes.io/projected/27c6a3fc-aff8-4c5f-a394-959078adbce1-kube-api-access-68kb7\") pod \"kserve-controller-manager-d6f4b65b8-kc6j5\" (UID: \"27c6a3fc-aff8-4c5f-a394-959078adbce1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:40.793857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.793782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5tn\" (UniqueName: \"kubernetes.io/projected/e537131c-098c-4ecc-a4f0-cf0b25d71930-kube-api-access-bd5tn\") pod \"llmisvc-controller-manager-777c6bd7d5-v5h7t\" (UID: \"e537131c-098c-4ecc-a4f0-cf0b25d71930\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:40.793857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.793850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e537131c-098c-4ecc-a4f0-cf0b25d71930-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v5h7t\" (UID: \"e537131c-098c-4ecc-a4f0-cf0b25d71930\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:40.794029 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:40.793983 2575 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 29 19:09:40.794064 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:40.794038 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e537131c-098c-4ecc-a4f0-cf0b25d71930-cert podName:e537131c-098c-4ecc-a4f0-cf0b25d71930 nodeName:}" failed. No retries permitted until 2026-07-29 19:09:41.294021195 +0000 UTC m=+562.311042269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e537131c-098c-4ecc-a4f0-cf0b25d71930-cert") pod "llmisvc-controller-manager-777c6bd7d5-v5h7t" (UID: "e537131c-098c-4ecc-a4f0-cf0b25d71930") : secret "llmisvc-webhook-server-cert" not found Jul 29 19:09:40.838385 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.838351 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t"] Jul 29 19:09:40.840053 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:40.840027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5tn\" (UniqueName: \"kubernetes.io/projected/e537131c-098c-4ecc-a4f0-cf0b25d71930-kube-api-access-bd5tn\") pod \"llmisvc-controller-manager-777c6bd7d5-v5h7t\" (UID: \"e537131c-098c-4ecc-a4f0-cf0b25d71930\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:41.197580 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.197540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27c6a3fc-aff8-4c5f-a394-959078adbce1-cert\") pod \"kserve-controller-manager-d6f4b65b8-kc6j5\" (UID: \"27c6a3fc-aff8-4c5f-a394-959078adbce1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:41.200277 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.200252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27c6a3fc-aff8-4c5f-a394-959078adbce1-cert\") pod \"kserve-controller-manager-d6f4b65b8-kc6j5\" (UID: \"27c6a3fc-aff8-4c5f-a394-959078adbce1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:41.298616 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.298567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e537131c-098c-4ecc-a4f0-cf0b25d71930-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v5h7t\" (UID: \"e537131c-098c-4ecc-a4f0-cf0b25d71930\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:41.301169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.301141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e537131c-098c-4ecc-a4f0-cf0b25d71930-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v5h7t\" (UID: \"e537131c-098c-4ecc-a4f0-cf0b25d71930\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:41.431941 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.431901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:41.499170 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.499127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:41.551156 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.550887 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h"] Jul 29 19:09:41.551156 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.551094 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" podUID="37c214f1-9315-4b57-9e3e-6082bad89146" containerName="discovery" containerID="cri-o://6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3" gracePeriod=30 Jul 29 19:09:41.610221 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.610196 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5"] Jul 29 19:09:41.662786 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:09:41.662753 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c6a3fc_aff8_4c5f_a394_959078adbce1.slice/crio-89226b4a0dfc220f85868a99fecf0b1ad7b5a34f42f7489e128e198b1d293746 WatchSource:0}: Error finding container 89226b4a0dfc220f85868a99fecf0b1ad7b5a34f42f7489e128e198b1d293746: Status 404 returned error can't find the container with id 89226b4a0dfc220f85868a99fecf0b1ad7b5a34f42f7489e128e198b1d293746 Jul 29 19:09:41.684816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.684756 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t"] Jul 29 19:09:41.687493 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:09:41.687467 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode537131c_098c_4ecc_a4f0_cf0b25d71930.slice/crio-3407aca2229304a01f12331ebd207002325608b406b4c225bb557d64606bdbf0 WatchSource:0}: Error finding container 3407aca2229304a01f12331ebd207002325608b406b4c225bb557d64606bdbf0: Status 404 returned error can't find the container with id 3407aca2229304a01f12331ebd207002325608b406b4c225bb557d64606bdbf0 Jul 29 19:09:41.814259 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.814232 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:41.906788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.906750 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-istio-token\") pod \"37c214f1-9315-4b57-9e3e-6082bad89146\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " Jul 29 19:09:41.906788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.906793 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n27mt\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-kube-api-access-n27mt\") pod \"37c214f1-9315-4b57-9e3e-6082bad89146\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " Jul 29 19:09:41.906988 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.906827 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c214f1-9315-4b57-9e3e-6082bad89146-local-certs\") pod \"37c214f1-9315-4b57-9e3e-6082bad89146\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " Jul 29 19:09:41.907027 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.907003 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-kubeconfig\") pod \"37c214f1-9315-4b57-9e3e-6082bad89146\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " Jul 29 19:09:41.907074 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.907058 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-cacerts\") pod \"37c214f1-9315-4b57-9e3e-6082bad89146\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " Jul 29 19:09:41.907163 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.907142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-ca-configmap\") pod \"37c214f1-9315-4b57-9e3e-6082bad89146\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " Jul 29 19:09:41.907294 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.907174 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-dns-cert\") pod \"37c214f1-9315-4b57-9e3e-6082bad89146\" (UID: \"37c214f1-9315-4b57-9e3e-6082bad89146\") " Jul 29 19:09:41.907593 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.907564 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "37c214f1-9315-4b57-9e3e-6082bad89146" (UID: "37c214f1-9315-4b57-9e3e-6082bad89146"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:09:41.909370 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.909327 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-kube-api-access-n27mt" (OuterVolumeSpecName: "kube-api-access-n27mt") pod "37c214f1-9315-4b57-9e3e-6082bad89146" (UID: "37c214f1-9315-4b57-9e3e-6082bad89146"). InnerVolumeSpecName "kube-api-access-n27mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:09:41.909457 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.909416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "37c214f1-9315-4b57-9e3e-6082bad89146" (UID: "37c214f1-9315-4b57-9e3e-6082bad89146"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:09:41.909614 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.909585 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c214f1-9315-4b57-9e3e-6082bad89146-local-certs" (OuterVolumeSpecName: "local-certs") pod "37c214f1-9315-4b57-9e3e-6082bad89146" (UID: "37c214f1-9315-4b57-9e3e-6082bad89146"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:09:41.909795 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.909611 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-istio-token" (OuterVolumeSpecName: "istio-token") pod "37c214f1-9315-4b57-9e3e-6082bad89146" (UID: "37c214f1-9315-4b57-9e3e-6082bad89146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:09:41.909795 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.909628 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-cacerts" (OuterVolumeSpecName: "cacerts") pod "37c214f1-9315-4b57-9e3e-6082bad89146" (UID: "37c214f1-9315-4b57-9e3e-6082bad89146"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:09:41.909795 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:41.909652 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "37c214f1-9315-4b57-9e3e-6082bad89146" (UID: "37c214f1-9315-4b57-9e3e-6082bad89146"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:09:42.008363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.008333 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-cacerts\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.008363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.008363 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-ca-configmap\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.008578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.008377 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-csr-dns-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.008578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.008390 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.008578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.008402 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27mt\" (UniqueName: \"kubernetes.io/projected/37c214f1-9315-4b57-9e3e-6082bad89146-kube-api-access-n27mt\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.008578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.008412 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c214f1-9315-4b57-9e3e-6082bad89146-local-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.008578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.008423 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37c214f1-9315-4b57-9e3e-6082bad89146-istio-kubeconfig\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.567388 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.567347 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" event={"ID":"e537131c-098c-4ecc-a4f0-cf0b25d71930","Type":"ContainerStarted","Data":"3407aca2229304a01f12331ebd207002325608b406b4c225bb557d64606bdbf0"} Jul 29 19:09:42.568678 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.568653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" event={"ID":"27c6a3fc-aff8-4c5f-a394-959078adbce1","Type":"ContainerStarted","Data":"89226b4a0dfc220f85868a99fecf0b1ad7b5a34f42f7489e128e198b1d293746"} Jul 29 19:09:42.569993 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.569971 2575 generic.go:356] "Generic (PLEG): container finished" podID="37c214f1-9315-4b57-9e3e-6082bad89146" containerID="6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3" exitCode=0 Jul 29 19:09:42.570083 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.570041 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" Jul 29 19:09:42.570134 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.570047 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" event={"ID":"37c214f1-9315-4b57-9e3e-6082bad89146","Type":"ContainerDied","Data":"6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3"} Jul 29 19:09:42.570175 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.570151 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h" event={"ID":"37c214f1-9315-4b57-9e3e-6082bad89146","Type":"ContainerDied","Data":"e85231008ec19e6e3f83a38140d8ea49db61621b8ed9f359d33cb0fbf58e3fa2"} Jul 29 19:09:42.570175 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.570171 2575 scope.go:117] "RemoveContainer" containerID="6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3" Jul 29 19:09:42.580522 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.580466 2575 scope.go:117] "RemoveContainer" containerID="6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3" Jul 29 19:09:42.580824 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:09:42.580804 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3\": container with ID starting with 6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3 not found: ID does not exist" containerID="6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3" Jul 29 19:09:42.580897 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.580833 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3"} err="failed to get container status \"6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3\": rpc error: code = NotFound desc = could not find container \"6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3\": container with ID starting with 6f7dac2a24f53481867cd4725332cf652a5ed62662a0fa1f5628f14250d087f3 not found: ID does not exist" Jul 29 19:09:42.627859 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.627830 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h"] Jul 29 19:09:42.630907 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:42.630883 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xj6h"] Jul 29 19:09:43.549216 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:43.549178 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37c214f1-9315-4b57-9e3e-6082bad89146" path="/var/lib/kubelet/pods/37c214f1-9315-4b57-9e3e-6082bad89146/volumes" Jul 29 19:09:44.069822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:44.069777 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:09:44.070257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:44.069869 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:09:44.070257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:44.069958 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:09:45.527324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:45.527293 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vz6ht" Jul 29 19:09:45.590078 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:45.590042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" event={"ID":"9eba4d7d-63f4-49a4-b90b-02c7888fc0c1","Type":"ContainerStarted","Data":"a40f605eb966773a5d1268d19c304bcc7a4736663cbb480ea0f3550c62509e39"} Jul 29 19:09:45.592031 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:45.591999 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" event={"ID":"27c6a3fc-aff8-4c5f-a394-959078adbce1","Type":"ContainerStarted","Data":"088eafa88b36dec94f8a9ecb43f447377f26f8155496e5f8abc7617766676bec"} Jul 29 19:09:45.592264 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:45.592246 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:09:45.613330 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:45.609556 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" podStartSLOduration=2.322089235 podStartE2EDuration="7.609530714s" totalImagesPullingTime="5.287441479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:45.608141521 +0000 UTC m=+566.625162627" watchObservedRunningTime="2026-07-29 19:09:45.609530714 +0000 UTC m=+566.626551810" Jul 29 19:09:45.625606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:45.625543 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" podStartSLOduration=2.3350934199999998 podStartE2EDuration="5.625498482s" totalImagesPullingTime="3.290405062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:45.624644619 +0000 UTC m=+566.641665718" watchObservedRunningTime="2026-07-29 19:09:45.625498482 +0000 UTC m=+566.642519595" Jul 29 19:09:45.629681 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:45.629653 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:46.598052 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:46.598010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" event={"ID":"e537131c-098c-4ecc-a4f0-cf0b25d71930","Type":"ContainerStarted","Data":"7c9658f9e35dac53235a297ef782159f9b94c1f1b67311b19d574d26004f13c6"} Jul 29 19:09:46.598427 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:46.598082 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:09:46.618935 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:46.618886 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" podStartSLOduration=2.34553196 podStartE2EDuration="6.61886785s" totalImagesPullingTime="4.27333589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:09:46.616618693 +0000 UTC m=+567.633639789" watchObservedRunningTime="2026-07-29 19:09:46.61886785 +0000 UTC m=+567.635888946" Jul 29 19:09:46.633622 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:46.633595 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:47.602970 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:47.602927 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:47.603910 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:47.603890 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9" Jul 29 19:09:48.546475 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:09:48.546444 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-czrkh" Jul 29 19:10:16.603980 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:16.603948 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kc6j5" Jul 29 19:10:17.605362 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:17.605330 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v5h7t" Jul 29 19:10:29.824141 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.824107 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kvz8d"] Jul 29 19:10:29.825607 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.824482 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37c214f1-9315-4b57-9e3e-6082bad89146" containerName="discovery" Jul 29 19:10:29.825607 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.824494 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="37c214f1-9315-4b57-9e3e-6082bad89146" containerName="discovery" Jul 29 19:10:29.825607 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.824591 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="37c214f1-9315-4b57-9e3e-6082bad89146" containerName="discovery" Jul 29 19:10:29.826650 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.826629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" Jul 29 19:10:29.829107 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.829081 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zvfpr\"" Jul 29 19:10:29.830125 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.830107 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 19:10:29.830341 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.830327 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 19:10:29.840338 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.840319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kvz8d"] Jul 29 19:10:29.910950 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:29.910925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbsr9\" (UniqueName: \"kubernetes.io/projected/19fbdff9-47ff-4f84-a427-2d929711a909-kube-api-access-tbsr9\") pod \"kuadrant-operator-catalog-kvz8d\" (UID: \"19fbdff9-47ff-4f84-a427-2d929711a909\") " pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" Jul 29 19:10:30.012147 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.012114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbsr9\" (UniqueName: \"kubernetes.io/projected/19fbdff9-47ff-4f84-a427-2d929711a909-kube-api-access-tbsr9\") pod \"kuadrant-operator-catalog-kvz8d\" (UID: \"19fbdff9-47ff-4f84-a427-2d929711a909\") " pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" Jul 29 19:10:30.022668 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.022641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbsr9\" (UniqueName: \"kubernetes.io/projected/19fbdff9-47ff-4f84-a427-2d929711a909-kube-api-access-tbsr9\") pod \"kuadrant-operator-catalog-kvz8d\" (UID: \"19fbdff9-47ff-4f84-a427-2d929711a909\") " pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" Jul 29 19:10:30.137435 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.137409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" Jul 29 19:10:30.265451 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.265423 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kvz8d"] Jul 29 19:10:30.267510 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:10:30.267481 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fbdff9_47ff_4f84_a427_2d929711a909.slice/crio-0b8edd79a6a7dffdac9f043f3478fe7179cf5b079cd365396c21bc9424ba65d8 WatchSource:0}: Error finding container 0b8edd79a6a7dffdac9f043f3478fe7179cf5b079cd365396c21bc9424ba65d8: Status 404 returned error can't find the container with id 0b8edd79a6a7dffdac9f043f3478fe7179cf5b079cd365396c21bc9424ba65d8 Jul 29 19:10:30.268907 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.268889 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:10:30.311177 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.311147 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kvz8d"] Jul 29 19:10:30.393128 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.393056 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qg4sz"] Jul 29 19:10:30.395910 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.395893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:30.405382 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.405354 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qg4sz"] Jul 29 19:10:30.517428 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.517393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkcm\" (UniqueName: \"kubernetes.io/projected/2caba4c5-0bf0-41b5-9c06-62356c6aee1e-kube-api-access-8rkcm\") pod \"kuadrant-operator-catalog-qg4sz\" (UID: \"2caba4c5-0bf0-41b5-9c06-62356c6aee1e\") " pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:30.618363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.618333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkcm\" (UniqueName: \"kubernetes.io/projected/2caba4c5-0bf0-41b5-9c06-62356c6aee1e-kube-api-access-8rkcm\") pod \"kuadrant-operator-catalog-qg4sz\" (UID: \"2caba4c5-0bf0-41b5-9c06-62356c6aee1e\") " pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:30.631384 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.631355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkcm\" (UniqueName: \"kubernetes.io/projected/2caba4c5-0bf0-41b5-9c06-62356c6aee1e-kube-api-access-8rkcm\") pod \"kuadrant-operator-catalog-qg4sz\" (UID: \"2caba4c5-0bf0-41b5-9c06-62356c6aee1e\") " pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:30.706065 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.705981 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:30.810674 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.810641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" event={"ID":"19fbdff9-47ff-4f84-a427-2d929711a909","Type":"ContainerStarted","Data":"0b8edd79a6a7dffdac9f043f3478fe7179cf5b079cd365396c21bc9424ba65d8"} Jul 29 19:10:30.834903 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:30.834876 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qg4sz"] Jul 29 19:10:30.836401 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:10:30.836375 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caba4c5_0bf0_41b5_9c06_62356c6aee1e.slice/crio-b2b53576b6009c8e1d49a9815afe7858773044fe1520e3ad80186ee4e10195f5 WatchSource:0}: Error finding container b2b53576b6009c8e1d49a9815afe7858773044fe1520e3ad80186ee4e10195f5: Status 404 returned error can't find the container with id b2b53576b6009c8e1d49a9815afe7858773044fe1520e3ad80186ee4e10195f5 Jul 29 19:10:31.817528 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:31.817481 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" event={"ID":"2caba4c5-0bf0-41b5-9c06-62356c6aee1e","Type":"ContainerStarted","Data":"b2b53576b6009c8e1d49a9815afe7858773044fe1520e3ad80186ee4e10195f5"} Jul 29 19:10:32.823323 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:32.823217 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" event={"ID":"2caba4c5-0bf0-41b5-9c06-62356c6aee1e","Type":"ContainerStarted","Data":"6b1970156af6ed5a3e0646c74cde04003028237c2b93a36a2480127d9ed92431"} Jul 29 19:10:32.824739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:32.824710 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" event={"ID":"19fbdff9-47ff-4f84-a427-2d929711a909","Type":"ContainerStarted","Data":"4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1"} Jul 29 19:10:32.824866 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:32.824716 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" podUID="19fbdff9-47ff-4f84-a427-2d929711a909" containerName="registry-server" containerID="cri-o://4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1" gracePeriod=2 Jul 29 19:10:32.842436 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:32.842392 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" podStartSLOduration=1.257423656 podStartE2EDuration="2.842377252s" totalImagesPullingTime="1.584953596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:10:32.840461593 +0000 UTC m=+613.857482691" watchObservedRunningTime="2026-07-29 19:10:32.842377252 +0000 UTC m=+613.859398347" Jul 29 19:10:32.858662 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:32.858621 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" podStartSLOduration=1.7055967669999998 podStartE2EDuration="3.858606621s" totalImagesPullingTime="2.153009854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:10:32.857990905 +0000 UTC m=+613.875011996" watchObservedRunningTime="2026-07-29 19:10:32.858606621 +0000 UTC m=+613.875627717" Jul 29 19:10:33.060608 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.060582 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" Jul 29 19:10:33.138438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.138414 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbsr9\" (UniqueName: \"kubernetes.io/projected/19fbdff9-47ff-4f84-a427-2d929711a909-kube-api-access-tbsr9\") pod \"19fbdff9-47ff-4f84-a427-2d929711a909\" (UID: \"19fbdff9-47ff-4f84-a427-2d929711a909\") " Jul 29 19:10:33.140560 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.140535 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fbdff9-47ff-4f84-a427-2d929711a909-kube-api-access-tbsr9" (OuterVolumeSpecName: "kube-api-access-tbsr9") pod "19fbdff9-47ff-4f84-a427-2d929711a909" (UID: "19fbdff9-47ff-4f84-a427-2d929711a909"). InnerVolumeSpecName "kube-api-access-tbsr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:10:33.239766 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.239736 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbsr9\" (UniqueName: \"kubernetes.io/projected/19fbdff9-47ff-4f84-a427-2d929711a909-kube-api-access-tbsr9\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:33.831369 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.831290 2575 generic.go:356] "Generic (PLEG): container finished" podID="19fbdff9-47ff-4f84-a427-2d929711a909" containerID="4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1" exitCode=0 Jul 29 19:10:33.831369 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.831337 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" Jul 29 19:10:33.831824 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.831371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" event={"ID":"19fbdff9-47ff-4f84-a427-2d929711a909","Type":"ContainerDied","Data":"4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1"} Jul 29 19:10:33.831824 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.831426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kvz8d" event={"ID":"19fbdff9-47ff-4f84-a427-2d929711a909","Type":"ContainerDied","Data":"0b8edd79a6a7dffdac9f043f3478fe7179cf5b079cd365396c21bc9424ba65d8"} Jul 29 19:10:33.831824 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.831451 2575 scope.go:117] "RemoveContainer" containerID="4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1" Jul 29 19:10:33.840352 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.840334 2575 scope.go:117] "RemoveContainer" containerID="4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1" Jul 29 19:10:33.840618 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:10:33.840601 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1\": container with ID starting with 4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1 not found: ID does not exist" containerID="4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1" Jul 29 19:10:33.840676 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.840625 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1"} err="failed to get container status \"4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1\": rpc error: code = NotFound desc = could not find container \"4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1\": container with ID starting with 4456e901e5e41070b3568965353b9e2533f5d1099ae19ea1656ce2ddf57e15b1 not found: ID does not exist" Jul 29 19:10:33.852590 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.852566 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kvz8d"] Jul 29 19:10:33.865384 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:33.865361 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kvz8d"] Jul 29 19:10:35.548824 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:35.548793 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19fbdff9-47ff-4f84-a427-2d929711a909" path="/var/lib/kubelet/pods/19fbdff9-47ff-4f84-a427-2d929711a909/volumes" Jul 29 19:10:40.706336 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:40.706298 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:40.706746 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:40.706390 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:40.727988 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:40.727954 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:40.884477 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:40.884448 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-qg4sz" Jul 29 19:10:44.978788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:44.978710 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9"] Jul 29 19:10:44.979140 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:44.979074 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fbdff9-47ff-4f84-a427-2d929711a909" containerName="registry-server" Jul 29 19:10:44.979140 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:44.979084 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fbdff9-47ff-4f84-a427-2d929711a909" containerName="registry-server" Jul 29 19:10:44.979209 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:44.979150 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="19fbdff9-47ff-4f84-a427-2d929711a909" containerName="registry-server" Jul 29 19:10:44.982189 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:44.982172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:44.985049 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:44.985024 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m7785\"" Jul 29 19:10:44.990655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:44.990632 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9"] Jul 29 19:10:45.042754 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.042716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcckj\" (UniqueName: \"kubernetes.io/projected/875ad67a-bb82-48fa-8e7d-671d1162e40a-kube-api-access-jcckj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.042897 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.042765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.042897 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.042794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.143712 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.143660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcckj\" (UniqueName: \"kubernetes.io/projected/875ad67a-bb82-48fa-8e7d-671d1162e40a-kube-api-access-jcckj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.143712 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.143717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.143920 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.143748 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.144098 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.144078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.144132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.144099 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.153158 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.153133 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcckj\" (UniqueName: \"kubernetes.io/projected/875ad67a-bb82-48fa-8e7d-671d1162e40a-kube-api-access-jcckj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.291948 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.291862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:45.423456 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.423426 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9"] Jul 29 19:10:45.426972 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:10:45.426941 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875ad67a_bb82_48fa_8e7d_671d1162e40a.slice/crio-57ccf06d639ee12265b3ac0f4a62f7b226f366a348acd4f3319a077a706b071d WatchSource:0}: Error finding container 57ccf06d639ee12265b3ac0f4a62f7b226f366a348acd4f3319a077a706b071d: Status 404 returned error can't find the container with id 57ccf06d639ee12265b3ac0f4a62f7b226f366a348acd4f3319a077a706b071d Jul 29 19:10:45.584549 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.584469 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp"] Jul 29 19:10:45.587956 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.587939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.595996 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.595955 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp"] Jul 29 19:10:45.647109 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.647082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829lw\" (UniqueName: \"kubernetes.io/projected/0c07a5c0-84a6-4145-88e4-81464dca4c60-kube-api-access-829lw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.647251 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.647229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.647308 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.647292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.748221 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.748191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.748357 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.748231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.748357 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.748270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829lw\" (UniqueName: \"kubernetes.io/projected/0c07a5c0-84a6-4145-88e4-81464dca4c60-kube-api-access-829lw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.748574 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.748555 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.748623 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.748602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.764124 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.764101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-829lw\" (UniqueName: \"kubernetes.io/projected/0c07a5c0-84a6-4145-88e4-81464dca4c60-kube-api-access-829lw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:45.894138 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.894108 2575 generic.go:356] "Generic (PLEG): container finished" podID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerID="8cc7ab37808d7d6f12c13ce47e9d01c02e99638fa378f6f935812d4aba794965" exitCode=0 Jul 29 19:10:45.894300 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.894199 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" event={"ID":"875ad67a-bb82-48fa-8e7d-671d1162e40a","Type":"ContainerDied","Data":"8cc7ab37808d7d6f12c13ce47e9d01c02e99638fa378f6f935812d4aba794965"} Jul 29 19:10:45.894300 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.894268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" event={"ID":"875ad67a-bb82-48fa-8e7d-671d1162e40a","Type":"ContainerStarted","Data":"57ccf06d639ee12265b3ac0f4a62f7b226f366a348acd4f3319a077a706b071d"} Jul 29 19:10:45.897132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:45.897110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:46.037804 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.037779 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp"] Jul 29 19:10:46.039125 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:10:46.039097 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c07a5c0_84a6_4145_88e4_81464dca4c60.slice/crio-8c3995c22551bacea6fc72004ceba6ed230ebf6d9fda6b798fe5bf20cb6b2ee7 WatchSource:0}: Error finding container 8c3995c22551bacea6fc72004ceba6ed230ebf6d9fda6b798fe5bf20cb6b2ee7: Status 404 returned error can't find the container with id 8c3995c22551bacea6fc72004ceba6ed230ebf6d9fda6b798fe5bf20cb6b2ee7 Jul 29 19:10:46.202760 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.202732 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl"] Jul 29 19:10:46.206283 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.206267 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.215428 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.215406 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl"] Jul 29 19:10:46.353501 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.353463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwct\" (UniqueName: \"kubernetes.io/projected/a0f9c84d-c11b-4e84-9393-3a072c48258a-kube-api-access-gvwct\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.353729 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.353505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.353729 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.353608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.454504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.454413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwct\" (UniqueName: \"kubernetes.io/projected/a0f9c84d-c11b-4e84-9393-3a072c48258a-kube-api-access-gvwct\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.454504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.454460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.454504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.454496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.454854 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.454832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.454897 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.454870 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.470383 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.470349 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwct\" (UniqueName: \"kubernetes.io/projected/a0f9c84d-c11b-4e84-9393-3a072c48258a-kube-api-access-gvwct\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.515301 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.515268 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:46.626966 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.626938 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g"] Jul 29 19:10:46.632616 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.632591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.641843 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.641814 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g"] Jul 29 19:10:46.682263 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.682231 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl"] Jul 29 19:10:46.742417 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:10:46.742353 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f9c84d_c11b_4e84_9393_3a072c48258a.slice/crio-973b9313db44d678620c8fa38a7a7cc689c6a37d2c639fe5d5655768a9d0257a WatchSource:0}: Error finding container 973b9313db44d678620c8fa38a7a7cc689c6a37d2c639fe5d5655768a9d0257a: Status 404 returned error can't find the container with id 973b9313db44d678620c8fa38a7a7cc689c6a37d2c639fe5d5655768a9d0257a Jul 29 19:10:46.756624 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.756599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.756752 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.756647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.756752 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.756672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzqv\" (UniqueName: \"kubernetes.io/projected/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-kube-api-access-bkzqv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.857723 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.857689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.857872 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.857740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.857872 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.857770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzqv\" (UniqueName: \"kubernetes.io/projected/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-kube-api-access-bkzqv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.858106 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.858083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.858169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.858119 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.866881 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.866861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzqv\" (UniqueName: \"kubernetes.io/projected/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-kube-api-access-bkzqv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:46.901934 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.901901 2575 generic.go:356] "Generic (PLEG): container finished" podID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerID="024ebe39974417686930ae58eb7dccc6763cd3eee763dd552d3d7b79e7824ce3" exitCode=0 Jul 29 19:10:46.902095 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.901977 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" event={"ID":"0c07a5c0-84a6-4145-88e4-81464dca4c60","Type":"ContainerDied","Data":"024ebe39974417686930ae58eb7dccc6763cd3eee763dd552d3d7b79e7824ce3"} Jul 29 19:10:46.902095 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.902028 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" event={"ID":"0c07a5c0-84a6-4145-88e4-81464dca4c60","Type":"ContainerStarted","Data":"8c3995c22551bacea6fc72004ceba6ed230ebf6d9fda6b798fe5bf20cb6b2ee7"} Jul 29 19:10:46.903892 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.903873 2575 generic.go:356] "Generic (PLEG): container finished" podID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerID="fad5c871a6522371607b2ecc00d61fe2f5f89f92f072ae167eb17a83952370d9" exitCode=0 Jul 29 19:10:46.903999 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.903948 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" event={"ID":"875ad67a-bb82-48fa-8e7d-671d1162e40a","Type":"ContainerDied","Data":"fad5c871a6522371607b2ecc00d61fe2f5f89f92f072ae167eb17a83952370d9"} Jul 29 19:10:46.905588 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.905563 2575 generic.go:356] "Generic (PLEG): container finished" podID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerID="bcad2219841222e12cdafa747bf87e1d925ad3158a00b2f1318d5000424f6010" exitCode=0 Jul 29 19:10:46.905725 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.905694 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" event={"ID":"a0f9c84d-c11b-4e84-9393-3a072c48258a","Type":"ContainerDied","Data":"bcad2219841222e12cdafa747bf87e1d925ad3158a00b2f1318d5000424f6010"} Jul 29 19:10:46.905789 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.905735 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" event={"ID":"a0f9c84d-c11b-4e84-9393-3a072c48258a","Type":"ContainerStarted","Data":"973b9313db44d678620c8fa38a7a7cc689c6a37d2c639fe5d5655768a9d0257a"} Jul 29 19:10:46.945101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:46.945084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:47.089567 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:47.089543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g"] Jul 29 19:10:47.113317 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:10:47.113291 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807cf65d_fbcf_4cfd_bfe8_6154f32fc057.slice/crio-c0526c6dbade8e556954920880abc7f0249066db66c82031e0b16b5b1959444e WatchSource:0}: Error finding container c0526c6dbade8e556954920880abc7f0249066db66c82031e0b16b5b1959444e: Status 404 returned error can't find the container with id c0526c6dbade8e556954920880abc7f0249066db66c82031e0b16b5b1959444e Jul 29 19:10:47.912115 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:47.912073 2575 generic.go:356] "Generic (PLEG): container finished" podID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerID="328cad873b3454cdaee284398982313240141688da098c27f6db601b81880c53" exitCode=0 Jul 29 19:10:47.912268 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:47.912152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" event={"ID":"875ad67a-bb82-48fa-8e7d-671d1162e40a","Type":"ContainerDied","Data":"328cad873b3454cdaee284398982313240141688da098c27f6db601b81880c53"} Jul 29 19:10:47.913628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:47.913607 2575 generic.go:356] "Generic (PLEG): container finished" podID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerID="c8b515f6a0127510d7e5bd3cfaa0bbbf2d1fc2573f6c42a773dd232230d2cc09" exitCode=0 Jul 29 19:10:47.913726 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:47.913685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" event={"ID":"807cf65d-fbcf-4cfd-bfe8-6154f32fc057","Type":"ContainerDied","Data":"c8b515f6a0127510d7e5bd3cfaa0bbbf2d1fc2573f6c42a773dd232230d2cc09"} Jul 29 19:10:47.913779 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:47.913728 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" event={"ID":"807cf65d-fbcf-4cfd-bfe8-6154f32fc057","Type":"ContainerStarted","Data":"c0526c6dbade8e556954920880abc7f0249066db66c82031e0b16b5b1959444e"} Jul 29 19:10:47.915429 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:47.915409 2575 generic.go:356] "Generic (PLEG): container finished" podID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerID="993d1496cff203d9d07e54c2f6791334579a52ac14c49a6ae10216f498e74e98" exitCode=0 Jul 29 19:10:47.915500 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:47.915433 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" event={"ID":"0c07a5c0-84a6-4145-88e4-81464dca4c60","Type":"ContainerDied","Data":"993d1496cff203d9d07e54c2f6791334579a52ac14c49a6ae10216f498e74e98"} Jul 29 19:10:48.933126 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:48.933099 2575 generic.go:356] "Generic (PLEG): container finished" podID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerID="7ea098413ac1c1ae4041bc7c94d120c4190fbfdca1ef2cbf9e55e2a046a9dba8" exitCode=0 Jul 29 19:10:48.933500 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:48.933183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" event={"ID":"a0f9c84d-c11b-4e84-9393-3a072c48258a","Type":"ContainerDied","Data":"7ea098413ac1c1ae4041bc7c94d120c4190fbfdca1ef2cbf9e55e2a046a9dba8"} Jul 29 19:10:48.935718 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:48.935693 2575 generic.go:356] "Generic (PLEG): container finished" podID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerID="e11df5100fed3e2447b4cb9c4688d62464ddf936faedc7816debdcccb4a6e088" exitCode=0 Jul 29 19:10:48.935826 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:48.935745 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" event={"ID":"0c07a5c0-84a6-4145-88e4-81464dca4c60","Type":"ContainerDied","Data":"e11df5100fed3e2447b4cb9c4688d62464ddf936faedc7816debdcccb4a6e088"} Jul 29 19:10:49.060461 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.060439 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:49.179787 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.179751 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcckj\" (UniqueName: \"kubernetes.io/projected/875ad67a-bb82-48fa-8e7d-671d1162e40a-kube-api-access-jcckj\") pod \"875ad67a-bb82-48fa-8e7d-671d1162e40a\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " Jul 29 19:10:49.179963 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.179792 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-util\") pod \"875ad67a-bb82-48fa-8e7d-671d1162e40a\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " Jul 29 19:10:49.179963 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.179820 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-bundle\") pod \"875ad67a-bb82-48fa-8e7d-671d1162e40a\" (UID: \"875ad67a-bb82-48fa-8e7d-671d1162e40a\") " Jul 29 19:10:49.180532 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.180483 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-bundle" (OuterVolumeSpecName: "bundle") pod "875ad67a-bb82-48fa-8e7d-671d1162e40a" (UID: "875ad67a-bb82-48fa-8e7d-671d1162e40a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:10:49.181780 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.181760 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875ad67a-bb82-48fa-8e7d-671d1162e40a-kube-api-access-jcckj" (OuterVolumeSpecName: "kube-api-access-jcckj") pod "875ad67a-bb82-48fa-8e7d-671d1162e40a" (UID: "875ad67a-bb82-48fa-8e7d-671d1162e40a"). InnerVolumeSpecName "kube-api-access-jcckj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:10:49.184938 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.184903 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-util" (OuterVolumeSpecName: "util") pod "875ad67a-bb82-48fa-8e7d-671d1162e40a" (UID: "875ad67a-bb82-48fa-8e7d-671d1162e40a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:10:49.280816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.280780 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcckj\" (UniqueName: \"kubernetes.io/projected/875ad67a-bb82-48fa-8e7d-671d1162e40a-kube-api-access-jcckj\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:49.280816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.280808 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-util\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:49.280816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.280817 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/875ad67a-bb82-48fa-8e7d-671d1162e40a-bundle\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:49.942076 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.942046 2575 generic.go:356] "Generic (PLEG): container finished" podID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerID="91ec039078b11aafb2707a3b570706a1a8836c0489695a4c15b953a8e8b252eb" exitCode=0 Jul 29 19:10:49.942499 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.942131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" event={"ID":"807cf65d-fbcf-4cfd-bfe8-6154f32fc057","Type":"ContainerDied","Data":"91ec039078b11aafb2707a3b570706a1a8836c0489695a4c15b953a8e8b252eb"} Jul 29 19:10:49.944202 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.944185 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" Jul 29 19:10:49.944202 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.944193 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9" event={"ID":"875ad67a-bb82-48fa-8e7d-671d1162e40a","Type":"ContainerDied","Data":"57ccf06d639ee12265b3ac0f4a62f7b226f366a348acd4f3319a077a706b071d"} Jul 29 19:10:49.944320 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.944225 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ccf06d639ee12265b3ac0f4a62f7b226f366a348acd4f3319a077a706b071d" Jul 29 19:10:49.946588 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.946570 2575 generic.go:356] "Generic (PLEG): container finished" podID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerID="a93847c6e4bb10a82c91896e18058df6bb533709ae65a313303f43f3c2f97c1e" exitCode=0 Jul 29 19:10:49.946690 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:49.946651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" event={"ID":"a0f9c84d-c11b-4e84-9393-3a072c48258a","Type":"ContainerDied","Data":"a93847c6e4bb10a82c91896e18058df6bb533709ae65a313303f43f3c2f97c1e"} Jul 29 19:10:50.143222 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.143199 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:50.290741 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.290666 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-829lw\" (UniqueName: \"kubernetes.io/projected/0c07a5c0-84a6-4145-88e4-81464dca4c60-kube-api-access-829lw\") pod \"0c07a5c0-84a6-4145-88e4-81464dca4c60\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " Jul 29 19:10:50.290741 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.290714 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-util\") pod \"0c07a5c0-84a6-4145-88e4-81464dca4c60\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " Jul 29 19:10:50.290915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.290755 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-bundle\") pod \"0c07a5c0-84a6-4145-88e4-81464dca4c60\" (UID: \"0c07a5c0-84a6-4145-88e4-81464dca4c60\") " Jul 29 19:10:50.291325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.291292 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-bundle" (OuterVolumeSpecName: "bundle") pod "0c07a5c0-84a6-4145-88e4-81464dca4c60" (UID: "0c07a5c0-84a6-4145-88e4-81464dca4c60"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:10:50.292701 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.292679 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c07a5c0-84a6-4145-88e4-81464dca4c60-kube-api-access-829lw" (OuterVolumeSpecName: "kube-api-access-829lw") pod "0c07a5c0-84a6-4145-88e4-81464dca4c60" (UID: "0c07a5c0-84a6-4145-88e4-81464dca4c60"). InnerVolumeSpecName "kube-api-access-829lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:10:50.296235 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.296211 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-util" (OuterVolumeSpecName: "util") pod "0c07a5c0-84a6-4145-88e4-81464dca4c60" (UID: "0c07a5c0-84a6-4145-88e4-81464dca4c60"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:10:50.391479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.391445 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829lw\" (UniqueName: \"kubernetes.io/projected/0c07a5c0-84a6-4145-88e4-81464dca4c60-kube-api-access-829lw\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:50.391479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.391475 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-util\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:50.391479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.391484 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c07a5c0-84a6-4145-88e4-81464dca4c60-bundle\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:50.953988 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.953960 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" Jul 29 19:10:50.954413 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.953988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp" event={"ID":"0c07a5c0-84a6-4145-88e4-81464dca4c60","Type":"ContainerDied","Data":"8c3995c22551bacea6fc72004ceba6ed230ebf6d9fda6b798fe5bf20cb6b2ee7"} Jul 29 19:10:50.954413 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.954031 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3995c22551bacea6fc72004ceba6ed230ebf6d9fda6b798fe5bf20cb6b2ee7" Jul 29 19:10:50.956047 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.956021 2575 generic.go:356] "Generic (PLEG): container finished" podID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerID="4cfe8668cdc0107e14aee532c6949b26e61daf7b2e2e81d1d1537b72bff93203" exitCode=0 Jul 29 19:10:50.956148 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:50.956111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" event={"ID":"807cf65d-fbcf-4cfd-bfe8-6154f32fc057","Type":"ContainerDied","Data":"4cfe8668cdc0107e14aee532c6949b26e61daf7b2e2e81d1d1537b72bff93203"} Jul 29 19:10:51.088975 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.088954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:51.196309 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.196273 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvwct\" (UniqueName: \"kubernetes.io/projected/a0f9c84d-c11b-4e84-9393-3a072c48258a-kube-api-access-gvwct\") pod \"a0f9c84d-c11b-4e84-9393-3a072c48258a\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " Jul 29 19:10:51.196467 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.196343 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-util\") pod \"a0f9c84d-c11b-4e84-9393-3a072c48258a\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " Jul 29 19:10:51.196467 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.196374 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-bundle\") pod \"a0f9c84d-c11b-4e84-9393-3a072c48258a\" (UID: \"a0f9c84d-c11b-4e84-9393-3a072c48258a\") " Jul 29 19:10:51.196927 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.196903 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-bundle" (OuterVolumeSpecName: "bundle") pod "a0f9c84d-c11b-4e84-9393-3a072c48258a" (UID: "a0f9c84d-c11b-4e84-9393-3a072c48258a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:10:51.198449 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.198423 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f9c84d-c11b-4e84-9393-3a072c48258a-kube-api-access-gvwct" (OuterVolumeSpecName: "kube-api-access-gvwct") pod "a0f9c84d-c11b-4e84-9393-3a072c48258a" (UID: "a0f9c84d-c11b-4e84-9393-3a072c48258a"). InnerVolumeSpecName "kube-api-access-gvwct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:10:51.201626 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.201596 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-util" (OuterVolumeSpecName: "util") pod "a0f9c84d-c11b-4e84-9393-3a072c48258a" (UID: "a0f9c84d-c11b-4e84-9393-3a072c48258a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:10:51.297255 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.297185 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-util\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:51.297255 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.297209 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f9c84d-c11b-4e84-9393-3a072c48258a-bundle\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:51.297255 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.297218 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvwct\" (UniqueName: \"kubernetes.io/projected/a0f9c84d-c11b-4e84-9393-3a072c48258a-kube-api-access-gvwct\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:51.962843 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.962803 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" Jul 29 19:10:51.962843 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.962822 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl" event={"ID":"a0f9c84d-c11b-4e84-9393-3a072c48258a","Type":"ContainerDied","Data":"973b9313db44d678620c8fa38a7a7cc689c6a37d2c639fe5d5655768a9d0257a"} Jul 29 19:10:51.963240 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:51.962867 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="973b9313db44d678620c8fa38a7a7cc689c6a37d2c639fe5d5655768a9d0257a" Jul 29 19:10:52.097730 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.097707 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:52.204495 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.204463 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkzqv\" (UniqueName: \"kubernetes.io/projected/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-kube-api-access-bkzqv\") pod \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " Jul 29 19:10:52.204650 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.204509 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-util\") pod \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " Jul 29 19:10:52.204650 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.204573 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-bundle\") pod \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\" (UID: \"807cf65d-fbcf-4cfd-bfe8-6154f32fc057\") " Jul 29 19:10:52.204968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.204938 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-bundle" (OuterVolumeSpecName: "bundle") pod "807cf65d-fbcf-4cfd-bfe8-6154f32fc057" (UID: "807cf65d-fbcf-4cfd-bfe8-6154f32fc057"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:10:52.206631 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.206603 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-kube-api-access-bkzqv" (OuterVolumeSpecName: "kube-api-access-bkzqv") pod "807cf65d-fbcf-4cfd-bfe8-6154f32fc057" (UID: "807cf65d-fbcf-4cfd-bfe8-6154f32fc057"). InnerVolumeSpecName "kube-api-access-bkzqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:10:52.209993 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.209975 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-util" (OuterVolumeSpecName: "util") pod "807cf65d-fbcf-4cfd-bfe8-6154f32fc057" (UID: "807cf65d-fbcf-4cfd-bfe8-6154f32fc057"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:10:52.305960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.305895 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-bundle\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:52.305960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.305918 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzqv\" (UniqueName: \"kubernetes.io/projected/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-kube-api-access-bkzqv\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:52.305960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.305928 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/807cf65d-fbcf-4cfd-bfe8-6154f32fc057-util\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:10:52.969940 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.969908 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" Jul 29 19:10:52.970337 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.969903 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g" event={"ID":"807cf65d-fbcf-4cfd-bfe8-6154f32fc057","Type":"ContainerDied","Data":"c0526c6dbade8e556954920880abc7f0249066db66c82031e0b16b5b1959444e"} Jul 29 19:10:52.970337 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:10:52.970040 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0526c6dbade8e556954920880abc7f0249066db66c82031e0b16b5b1959444e" Jul 29 19:11:01.437691 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.437653 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7"] Jul 29 19:11:01.438037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438005 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerName="extract" Jul 29 19:11:01.438037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438016 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerName="extract" Jul 29 19:11:01.438037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438031 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerName="util" Jul 29 19:11:01.438037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438037 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerName="util" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438046 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerName="pull" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438052 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerName="pull" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438058 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerName="util" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438063 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerName="util" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438068 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerName="util" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438074 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerName="util" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438079 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerName="pull" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438084 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerName="pull" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438090 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerName="extract" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438095 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerName="extract" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438101 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerName="pull" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438106 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerName="pull" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438114 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerName="util" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438119 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerName="util" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438126 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerName="extract" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438130 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerName="extract" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438138 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerName="extract" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438143 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerName="extract" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438152 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerName="pull" Jul 29 19:11:01.438171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438156 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerName="pull" Jul 29 19:11:01.438727 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438207 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="875ad67a-bb82-48fa-8e7d-671d1162e40a" containerName="extract" Jul 29 19:11:01.438727 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438215 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f9c84d-c11b-4e84-9393-3a072c48258a" containerName="extract" Jul 29 19:11:01.438727 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438222 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="807cf65d-fbcf-4cfd-bfe8-6154f32fc057" containerName="extract" Jul 29 19:11:01.438727 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.438229 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c07a5c0-84a6-4145-88e4-81464dca4c60" containerName="extract" Jul 29 19:11:01.441410 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.441393 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:01.443744 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.443723 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8zr2p\"" Jul 29 19:11:01.462344 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.462323 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7"] Jul 29 19:11:01.587956 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.587915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hds5n\" (UniqueName: \"kubernetes.io/projected/ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5-kube-api-access-hds5n\") pod \"limitador-operator-controller-manager-859d7b78bd-2h7v7\" (UID: \"ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:01.689357 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.689277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hds5n\" (UniqueName: \"kubernetes.io/projected/ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5-kube-api-access-hds5n\") pod \"limitador-operator-controller-manager-859d7b78bd-2h7v7\" (UID: \"ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:01.697168 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.697140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hds5n\" (UniqueName: \"kubernetes.io/projected/ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5-kube-api-access-hds5n\") pod \"limitador-operator-controller-manager-859d7b78bd-2h7v7\" (UID: \"ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:01.751698 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.751670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:01.891615 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:01.891594 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7"] Jul 29 19:11:01.893498 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:11:01.893470 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce699d8e_e1f6_43ac_a9ec_8325fe1d1ca5.slice/crio-7f1d3914c01fad31c8fc7f0a0235a298c205a0d4c20bb30f92f86db04f0ffd5a WatchSource:0}: Error finding container 7f1d3914c01fad31c8fc7f0a0235a298c205a0d4c20bb30f92f86db04f0ffd5a: Status 404 returned error can't find the container with id 7f1d3914c01fad31c8fc7f0a0235a298c205a0d4c20bb30f92f86db04f0ffd5a Jul 29 19:11:02.017101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:02.017014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" event={"ID":"ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5","Type":"ContainerStarted","Data":"7f1d3914c01fad31c8fc7f0a0235a298c205a0d4c20bb30f92f86db04f0ffd5a"} Jul 29 19:11:04.030505 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:04.030463 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" event={"ID":"ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5","Type":"ContainerStarted","Data":"b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70"} Jul 29 19:11:04.030924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:04.030571 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:04.053070 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:04.053023 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" podStartSLOduration=1.378643942 podStartE2EDuration="3.053010741s" totalImagesPullingTime="1.674366799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:04.051624442 +0000 UTC m=+645.068645539" watchObservedRunningTime="2026-07-29 19:11:04.053010741 +0000 UTC m=+645.070031837" Jul 29 19:11:05.697558 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:05.697509 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl"] Jul 29 19:11:05.701307 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:05.701287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" Jul 29 19:11:05.704553 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:05.704534 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j9sd2\"" Jul 29 19:11:05.724961 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:05.724933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59jgt\" (UniqueName: \"kubernetes.io/projected/ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e-kube-api-access-59jgt\") pod \"authorino-operator-75cc8d5cb7-z2vvl\" (UID: \"ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" Jul 29 19:11:05.725960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:05.725943 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl"] Jul 29 19:11:05.825640 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:05.825605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59jgt\" (UniqueName: \"kubernetes.io/projected/ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e-kube-api-access-59jgt\") pod \"authorino-operator-75cc8d5cb7-z2vvl\" (UID: \"ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" Jul 29 19:11:05.845020 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:05.844988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59jgt\" (UniqueName: \"kubernetes.io/projected/ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e-kube-api-access-59jgt\") pod \"authorino-operator-75cc8d5cb7-z2vvl\" (UID: \"ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" Jul 29 19:11:06.011367 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:06.011283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" Jul 29 19:11:06.155025 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:06.154996 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl"] Jul 29 19:11:06.157178 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:11:06.157151 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad186fd6_2d4c_41e4_9d6b_1a0922a12f2e.slice/crio-62ba117d004ec5049b897f35bc01203f3d5a1c76a9e222a3af98b8ecfdf73c50 WatchSource:0}: Error finding container 62ba117d004ec5049b897f35bc01203f3d5a1c76a9e222a3af98b8ecfdf73c50: Status 404 returned error can't find the container with id 62ba117d004ec5049b897f35bc01203f3d5a1c76a9e222a3af98b8ecfdf73c50 Jul 29 19:11:07.048737 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:07.048661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" event={"ID":"ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e","Type":"ContainerStarted","Data":"62ba117d004ec5049b897f35bc01203f3d5a1c76a9e222a3af98b8ecfdf73c50"} Jul 29 19:11:09.061342 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:09.061305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" event={"ID":"ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e","Type":"ContainerStarted","Data":"45e20d8bcdc3bed5b3bfa869f1a4770b4c900672f7ceea9bb9e751306e3d1b3a"} Jul 29 19:11:09.061759 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:09.061435 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" Jul 29 19:11:09.084195 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:09.084148 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" podStartSLOduration=1.8683375469999999 podStartE2EDuration="4.08413588s" totalImagesPullingTime="2.215798333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:09.081703274 +0000 UTC m=+650.098724371" watchObservedRunningTime="2026-07-29 19:11:09.08413588 +0000 UTC m=+650.101156975" Jul 29 19:11:10.832630 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.832599 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk"] Jul 29 19:11:10.834986 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.834968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:10.838241 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.838220 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-879x4\"" Jul 29 19:11:10.861163 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.861139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk"] Jul 29 19:11:10.865748 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.865720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d056195-9b33-4763-81e2-d695b90e05de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" (UID: \"2d056195-9b33-4763-81e2-d695b90e05de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:10.865888 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.865870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kffmw\" (UniqueName: \"kubernetes.io/projected/2d056195-9b33-4763-81e2-d695b90e05de-kube-api-access-kffmw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" (UID: \"2d056195-9b33-4763-81e2-d695b90e05de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:10.966257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.966223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kffmw\" (UniqueName: \"kubernetes.io/projected/2d056195-9b33-4763-81e2-d695b90e05de-kube-api-access-kffmw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" (UID: \"2d056195-9b33-4763-81e2-d695b90e05de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:10.966411 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.966302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d056195-9b33-4763-81e2-d695b90e05de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" (UID: \"2d056195-9b33-4763-81e2-d695b90e05de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:10.966669 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.966653 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d056195-9b33-4763-81e2-d695b90e05de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" (UID: \"2d056195-9b33-4763-81e2-d695b90e05de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:10.980204 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:10.980173 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kffmw\" (UniqueName: \"kubernetes.io/projected/2d056195-9b33-4763-81e2-d695b90e05de-kube-api-access-kffmw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" (UID: \"2d056195-9b33-4763-81e2-d695b90e05de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:11.144857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:11.144813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:11.285324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:11.285297 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk"] Jul 29 19:11:11.288575 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:11:11.288542 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d056195_9b33_4763_81e2_d695b90e05de.slice/crio-c5ce7c524b16db5b80128bbafd02754956ec3ab47c5e2908bc48ae559b41b799 WatchSource:0}: Error finding container c5ce7c524b16db5b80128bbafd02754956ec3ab47c5e2908bc48ae559b41b799: Status 404 returned error can't find the container with id c5ce7c524b16db5b80128bbafd02754956ec3ab47c5e2908bc48ae559b41b799 Jul 29 19:11:12.078376 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:12.078339 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" event={"ID":"2d056195-9b33-4763-81e2-d695b90e05de","Type":"ContainerStarted","Data":"c5ce7c524b16db5b80128bbafd02754956ec3ab47c5e2908bc48ae559b41b799"} Jul 29 19:11:15.038148 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:15.038113 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:19.121800 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:19.121759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" event={"ID":"2d056195-9b33-4763-81e2-d695b90e05de","Type":"ContainerStarted","Data":"ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a"} Jul 29 19:11:19.122165 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:19.121913 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:19.146889 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:19.146835 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" podStartSLOduration=1.573713809 podStartE2EDuration="9.146817248s" totalImagesPullingTime="7.573103439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:19.145718681 +0000 UTC m=+660.162739782" watchObservedRunningTime="2026-07-29 19:11:19.146817248 +0000 UTC m=+660.163838413" Jul 29 19:11:20.069283 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:20.069240 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2vvl" Jul 29 19:11:21.595627 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.595593 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn"] Jul 29 19:11:21.600354 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.600333 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:21.603021 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.602994 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 29 19:11:21.603021 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.603012 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 29 19:11:21.603184 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.603010 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m7785\"" Jul 29 19:11:21.609760 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.609732 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn"] Jul 29 19:11:21.662205 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.662173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:21.662371 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.662242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpbp\" (UniqueName: \"kubernetes.io/projected/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-kube-api-access-kkpbp\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:21.662371 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.662308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:21.763240 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.763202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpbp\" (UniqueName: \"kubernetes.io/projected/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-kube-api-access-kkpbp\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:21.763406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.763250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:21.763406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.763289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:21.763476 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:21.763446 2575 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 29 19:11:21.763538 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:21.763509 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-plugin-serving-cert podName:dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764 nodeName:}" failed. No retries permitted until 2026-07-29 19:11:22.263491042 +0000 UTC m=+663.280512116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-zfxwn" (UID: "dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764") : secret "plugin-serving-cert" not found Jul 29 19:11:21.763985 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.763966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:21.774698 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:21.774675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpbp\" (UniqueName: \"kubernetes.io/projected/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-kube-api-access-kkpbp\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:22.269378 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:22.269329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:22.271660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:22.271643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfxwn\" (UID: \"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:22.511561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:22.511505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" Jul 29 19:11:22.646892 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:22.646788 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn"] Jul 29 19:11:22.652016 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:11:22.651990 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfbd83d9_4ccb_4dd1_82fd_3ce127e4f764.slice/crio-2e26d04b6ba28ab751b441a79b1d3e9f4ea8da4bdf0e7c370614a8f27b48396f WatchSource:0}: Error finding container 2e26d04b6ba28ab751b441a79b1d3e9f4ea8da4bdf0e7c370614a8f27b48396f: Status 404 returned error can't find the container with id 2e26d04b6ba28ab751b441a79b1d3e9f4ea8da4bdf0e7c370614a8f27b48396f Jul 29 19:11:23.145092 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:23.145051 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" event={"ID":"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764","Type":"ContainerStarted","Data":"2e26d04b6ba28ab751b441a79b1d3e9f4ea8da4bdf0e7c370614a8f27b48396f"} Jul 29 19:11:30.130174 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:30.130140 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:31.843012 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:31.842979 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh"] Jul 29 19:11:31.849992 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:31.849962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:31.878340 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:31.878307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh"] Jul 29 19:11:31.960643 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:31.960609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nvnt\" (UniqueName: \"kubernetes.io/projected/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-kube-api-access-6nvnt\") pod \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" (UID: \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:31.960835 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:31.960652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" (UID: \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:32.061782 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.061741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nvnt\" (UniqueName: \"kubernetes.io/projected/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-kube-api-access-6nvnt\") pod \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" (UID: \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:32.061974 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.061795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" (UID: \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:32.062330 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.062301 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" (UID: \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:32.075986 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.075961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nvnt\" (UniqueName: \"kubernetes.io/projected/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-kube-api-access-6nvnt\") pod \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" (UID: \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:32.163234 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.163182 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:32.196811 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.196776 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh"] Jul 29 19:11:32.205093 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.205067 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh"] Jul 29 19:11:32.223389 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.223344 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk"] Jul 29 19:11:32.224132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.223961 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" podUID="2d056195-9b33-4763-81e2-d695b90e05de" containerName="manager" containerID="cri-o://ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a" gracePeriod=2 Jul 29 19:11:32.226945 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.226894 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk"] Jul 29 19:11:32.233086 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:32.233049 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="2d056195-9b33-4763-81e2-d695b90e05de" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:32.235613 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.235589 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h"] Jul 29 19:11:32.236267 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.236250 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d056195-9b33-4763-81e2-d695b90e05de" containerName="manager" Jul 29 19:11:32.236375 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.236356 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d056195-9b33-4763-81e2-d695b90e05de" containerName="manager" Jul 29 19:11:32.236483 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.236461 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d056195-9b33-4763-81e2-d695b90e05de" containerName="manager" Jul 29 19:11:32.242653 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.242629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:32.253759 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.253737 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h"] Jul 29 19:11:32.256859 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.256833 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7"] Jul 29 19:11:32.257113 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.257054 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" containerName="manager" containerID="cri-o://b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70" gracePeriod=2 Jul 29 19:11:32.265353 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.265186 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb"] Jul 29 19:11:32.271394 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.271369 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7"] Jul 29 19:11:32.271503 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.271493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:32.276906 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:32.276872 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="2d056195-9b33-4763-81e2-d695b90e05de" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:32.283725 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.283652 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb"] Jul 29 19:11:32.312388 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:32.312265 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2h7v7\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:32.314571 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:32.314528 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="2d056195-9b33-4763-81e2-d695b90e05de" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:32.363320 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.363286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slsv2\" (UniqueName: \"kubernetes.io/projected/37983937-af76-4800-a48b-3250ee6b954b-kube-api-access-slsv2\") pod \"kuadrant-operator-controller-manager-55467d9f7-jb77h\" (UID: \"37983937-af76-4800-a48b-3250ee6b954b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:32.363475 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.363330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk5jz\" (UniqueName: \"kubernetes.io/projected/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-kube-api-access-tk5jz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57wbb\" (UID: \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:32.363475 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.363406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37983937-af76-4800-a48b-3250ee6b954b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-jb77h\" (UID: \"37983937-af76-4800-a48b-3250ee6b954b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:32.363635 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.363553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57wbb\" (UID: \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:32.464380 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.464291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57wbb\" (UID: \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:32.464380 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.464371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slsv2\" (UniqueName: \"kubernetes.io/projected/37983937-af76-4800-a48b-3250ee6b954b-kube-api-access-slsv2\") pod \"kuadrant-operator-controller-manager-55467d9f7-jb77h\" (UID: \"37983937-af76-4800-a48b-3250ee6b954b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:32.464619 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.464391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk5jz\" (UniqueName: \"kubernetes.io/projected/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-kube-api-access-tk5jz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57wbb\" (UID: \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:32.464619 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.464435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37983937-af76-4800-a48b-3250ee6b954b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-jb77h\" (UID: \"37983937-af76-4800-a48b-3250ee6b954b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:32.464744 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.464720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57wbb\" (UID: \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:32.464841 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.464821 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37983937-af76-4800-a48b-3250ee6b954b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-jb77h\" (UID: \"37983937-af76-4800-a48b-3250ee6b954b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:32.476806 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.476780 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slsv2\" (UniqueName: \"kubernetes.io/projected/37983937-af76-4800-a48b-3250ee6b954b-kube-api-access-slsv2\") pod \"kuadrant-operator-controller-manager-55467d9f7-jb77h\" (UID: \"37983937-af76-4800-a48b-3250ee6b954b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:32.476942 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.476930 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk5jz\" (UniqueName: \"kubernetes.io/projected/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-kube-api-access-tk5jz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-57wbb\" (UID: \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:32.646571 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.646537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:32.653434 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:32.653409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:45.780088 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.780045 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:45.782733 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:45.782703 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2h7v7\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:45.807629 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.807585 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:45.810745 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:45.810716 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2h7v7\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:45.814973 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:45.814936 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="2d056195-9b33-4763-81e2-d695b90e05de" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:45.848349 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.848320 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h"] Jul 29 19:11:45.851939 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:11:45.851902 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37983937_af76_4800_a48b_3250ee6b954b.slice/crio-ae5592db928bc088c3d33e3a1ea347db0ea939828edbc9e468a561732dbd6201 WatchSource:0}: Error finding container ae5592db928bc088c3d33e3a1ea347db0ea939828edbc9e468a561732dbd6201: Status 404 returned error can't find the container with id ae5592db928bc088c3d33e3a1ea347db0ea939828edbc9e468a561732dbd6201 Jul 29 19:11:45.888922 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.888897 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d056195-9b33-4763-81e2-d695b90e05de-extensions-socket-volume\") pod \"2d056195-9b33-4763-81e2-d695b90e05de\" (UID: \"2d056195-9b33-4763-81e2-d695b90e05de\") " Jul 29 19:11:45.889012 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.888947 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kffmw\" (UniqueName: \"kubernetes.io/projected/2d056195-9b33-4763-81e2-d695b90e05de-kube-api-access-kffmw\") pod \"2d056195-9b33-4763-81e2-d695b90e05de\" (UID: \"2d056195-9b33-4763-81e2-d695b90e05de\") " Jul 29 19:11:45.889918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.889028 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hds5n\" (UniqueName: \"kubernetes.io/projected/ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5-kube-api-access-hds5n\") pod \"ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5\" (UID: \"ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5\") " Jul 29 19:11:45.889918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.889300 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d056195-9b33-4763-81e2-d695b90e05de-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2d056195-9b33-4763-81e2-d695b90e05de" (UID: "2d056195-9b33-4763-81e2-d695b90e05de"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:11:45.891210 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.891187 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5-kube-api-access-hds5n" (OuterVolumeSpecName: "kube-api-access-hds5n") pod "ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" (UID: "ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5"). InnerVolumeSpecName "kube-api-access-hds5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:11:45.891418 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.891405 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d056195-9b33-4763-81e2-d695b90e05de-kube-api-access-kffmw" (OuterVolumeSpecName: "kube-api-access-kffmw") pod "2d056195-9b33-4763-81e2-d695b90e05de" (UID: "2d056195-9b33-4763-81e2-d695b90e05de"). InnerVolumeSpecName "kube-api-access-kffmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:11:45.903241 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:11:45.903203 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dee8e26_724b_4dc7_b4dd_a564b7f165ce.slice/crio-a8ae3aeffed583a275a767f59ec594f1a6a916970cb33f4fbd2dc16e1fdb1e5d WatchSource:0}: Error finding container a8ae3aeffed583a275a767f59ec594f1a6a916970cb33f4fbd2dc16e1fdb1e5d: Status 404 returned error can't find the container with id a8ae3aeffed583a275a767f59ec594f1a6a916970cb33f4fbd2dc16e1fdb1e5d Jul 29 19:11:45.908439 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.908417 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb"] Jul 29 19:11:45.990386 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.990362 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kffmw\" (UniqueName: \"kubernetes.io/projected/2d056195-9b33-4763-81e2-d695b90e05de-kube-api-access-kffmw\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:11:45.990386 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.990385 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hds5n\" (UniqueName: \"kubernetes.io/projected/ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5-kube-api-access-hds5n\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:11:45.990509 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:45.990396 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d056195-9b33-4763-81e2-d695b90e05de-extensions-socket-volume\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:11:46.289793 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.289702 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" event={"ID":"1dee8e26-724b-4dc7-b4dd-a564b7f165ce","Type":"ContainerStarted","Data":"610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4"} Jul 29 19:11:46.289793 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.289766 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:46.289793 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.289789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" event={"ID":"1dee8e26-724b-4dc7-b4dd-a564b7f165ce","Type":"ContainerStarted","Data":"a8ae3aeffed583a275a767f59ec594f1a6a916970cb33f4fbd2dc16e1fdb1e5d"} Jul 29 19:11:46.291328 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.291300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" event={"ID":"37983937-af76-4800-a48b-3250ee6b954b","Type":"ContainerStarted","Data":"3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f"} Jul 29 19:11:46.291448 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.291352 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" event={"ID":"37983937-af76-4800-a48b-3250ee6b954b","Type":"ContainerStarted","Data":"ae5592db928bc088c3d33e3a1ea347db0ea939828edbc9e468a561732dbd6201"} Jul 29 19:11:46.291448 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.291377 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:46.292750 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.292722 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" event={"ID":"dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764","Type":"ContainerStarted","Data":"03f7a3fd5a8b2293f9134664f5adb46c0cbcd0f77d9c042f1520ae9b34c0503b"} Jul 29 19:11:46.294020 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.294000 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d056195-9b33-4763-81e2-d695b90e05de" containerID="ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a" exitCode=0 Jul 29 19:11:46.294131 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.294031 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:46.294131 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.294065 2575 scope.go:117] "RemoveContainer" containerID="ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a" Jul 29 19:11:46.295464 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.295434 2575 generic.go:356] "Generic (PLEG): container finished" podID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" containerID="b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70" exitCode=0 Jul 29 19:11:46.295589 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.295481 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:46.303552 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.303505 2575 scope.go:117] "RemoveContainer" containerID="ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a" Jul 29 19:11:46.303831 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:46.303803 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a\": container with ID starting with ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a not found: ID does not exist" containerID="ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a" Jul 29 19:11:46.303906 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.303846 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a"} err="failed to get container status \"ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a\": rpc error: code = NotFound desc = could not find container \"ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a\": container with ID starting with ad9487af62f2b1945c0e6c59d5e78f72556a9fc4469113df1f7a78588481fe8a not found: ID does not exist" Jul 29 19:11:46.303906 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.303872 2575 scope.go:117] "RemoveContainer" containerID="b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70" Jul 29 19:11:46.311574 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.311558 2575 scope.go:117] "RemoveContainer" containerID="b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70" Jul 29 19:11:46.311832 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:46.311813 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70\": container with ID starting with b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70 not found: ID does not exist" containerID="b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70" Jul 29 19:11:46.311896 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.311841 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70"} err="failed to get container status \"b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70\": rpc error: code = NotFound desc = could not find container \"b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70\": container with ID starting with b17bfe1c79f31d89840dfd22e40253c79b6c9297096f6ee84c5b754a2dce7b70 not found: ID does not exist" Jul 29 19:11:46.313223 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.313181 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" podStartSLOduration=14.313166039 podStartE2EDuration="14.313166039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:46.313069435 +0000 UTC m=+687.330090530" watchObservedRunningTime="2026-07-29 19:11:46.313166039 +0000 UTC m=+687.330187134" Jul 29 19:11:46.315478 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:46.315453 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2h7v7\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:46.317564 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:46.317540 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="2d056195-9b33-4763-81e2-d695b90e05de" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:46.319783 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:46.319761 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dd6tk\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="2d056195-9b33-4763-81e2-d695b90e05de" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dd6tk" Jul 29 19:11:46.351406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.351355 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" podStartSLOduration=14.351341353 podStartE2EDuration="14.351341353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:46.346952512 +0000 UTC m=+687.363973609" watchObservedRunningTime="2026-07-29 19:11:46.351341353 +0000 UTC m=+687.368362448" Jul 29 19:11:46.368116 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:46.368070 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfxwn" podStartSLOduration=2.309805713 podStartE2EDuration="25.368057678s" totalImagesPullingTime="23.058251965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:46.366849049 +0000 UTC m=+687.383870146" watchObservedRunningTime="2026-07-29 19:11:46.368057678 +0000 UTC m=+687.385078773" Jul 29 19:11:46.369268 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:46.369241 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2h7v7\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2h7v7" Jul 29 19:11:47.551640 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:47.551598 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d056195-9b33-4763-81e2-d695b90e05de" path="/var/lib/kubelet/pods/2d056195-9b33-4763-81e2-d695b90e05de/volumes" Jul 29 19:11:47.552739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:47.552718 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" path="/var/lib/kubelet/pods/ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5/volumes" Jul 29 19:11:48.207505 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:48.207465 2575 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-dtmqh_kuadrant-system_0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89_0(b985996cca5704df9053bfd2ed72f198995644348ef79ea7b2c9c4d2e349b7aa): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-dtmqh to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 29 19:11:48.207620 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:48.207552 2575 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-dtmqh_kuadrant-system_0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89_0(b985996cca5704df9053bfd2ed72f198995644348ef79ea7b2c9c4d2e349b7aa): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-dtmqh to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:48.308320 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:48.308290 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:48.311972 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:48.311938 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:48.314901 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:48.314882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:48.317656 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:48.317630 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:48.412168 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:48.412145 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nvnt\" (UniqueName: \"kubernetes.io/projected/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-kube-api-access-6nvnt\") pod \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\" (UID: \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\") " Jul 29 19:11:48.412261 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:48.412181 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-extensions-socket-volume\") pod \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\" (UID: \"0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89\") " Jul 29 19:11:48.412496 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:48.412477 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89" (UID: "0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:11:48.414208 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:48.414182 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-kube-api-access-6nvnt" (OuterVolumeSpecName: "kube-api-access-6nvnt") pod "0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89" (UID: "0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89"). InnerVolumeSpecName "kube-api-access-6nvnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:11:48.513099 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:48.513028 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nvnt\" (UniqueName: \"kubernetes.io/projected/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-kube-api-access-6nvnt\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:11:48.513099 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:48.513052 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89-extensions-socket-volume\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:11:49.313562 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:49.313530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:49.316032 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:49.316000 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:49.321695 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:49.321670 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:49.548225 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:49.548192 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dtmqh\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dtmqh" Jul 29 19:11:49.549167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:49.549147 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89" path="/var/lib/kubelet/pods/0c9c9cbb-ef0f-4b7d-a2b5-7224b0a1ce89/volumes" Jul 29 19:11:57.305433 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.305398 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:57.305995 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.305584 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:11:57.375480 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.375444 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb"] Jul 29 19:11:57.375712 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.375687 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" podUID="1dee8e26-724b-4dc7-b4dd-a564b7f165ce" containerName="manager" containerID="cri-o://610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4" gracePeriod=10 Jul 29 19:11:57.571757 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.571691 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2"] Jul 29 19:11:57.572068 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.572056 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" containerName="manager" Jul 29 19:11:57.572115 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.572069 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" containerName="manager" Jul 29 19:11:57.572153 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.572135 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce699d8e-e1f6-43ac-a9ec-8325fe1d1ca5" containerName="manager" Jul 29 19:11:57.574595 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.574575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:57.590073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.590048 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2"] Jul 29 19:11:57.591920 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.591893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzfh\" (UniqueName: \"kubernetes.io/projected/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-kube-api-access-6zzfh\") pod \"kuadrant-operator-controller-manager-86878c769f-wd4r2\" (UID: \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:57.592266 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.592245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wd4r2\" (UID: \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:57.628940 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.628920 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:57.692640 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.692609 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk5jz\" (UniqueName: \"kubernetes.io/projected/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-kube-api-access-tk5jz\") pod \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\" (UID: \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\") " Jul 29 19:11:57.692799 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.692663 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-extensions-socket-volume\") pod \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\" (UID: \"1dee8e26-724b-4dc7-b4dd-a564b7f165ce\") " Jul 29 19:11:57.692858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.692807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wd4r2\" (UID: \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:57.692913 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.692871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzfh\" (UniqueName: \"kubernetes.io/projected/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-kube-api-access-6zzfh\") pod \"kuadrant-operator-controller-manager-86878c769f-wd4r2\" (UID: \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:57.693111 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.693087 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1dee8e26-724b-4dc7-b4dd-a564b7f165ce" (UID: "1dee8e26-724b-4dc7-b4dd-a564b7f165ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:11:57.693224 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.693202 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wd4r2\" (UID: \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:57.694466 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.694444 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-kube-api-access-tk5jz" (OuterVolumeSpecName: "kube-api-access-tk5jz") pod "1dee8e26-724b-4dc7-b4dd-a564b7f165ce" (UID: "1dee8e26-724b-4dc7-b4dd-a564b7f165ce"). InnerVolumeSpecName "kube-api-access-tk5jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:11:57.701733 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.701715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzfh\" (UniqueName: \"kubernetes.io/projected/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-kube-api-access-6zzfh\") pod \"kuadrant-operator-controller-manager-86878c769f-wd4r2\" (UID: \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:57.793827 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.793801 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk5jz\" (UniqueName: \"kubernetes.io/projected/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-kube-api-access-tk5jz\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:11:57.793827 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.793827 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dee8e26-724b-4dc7-b4dd-a564b7f165ce-extensions-socket-volume\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:11:57.885073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:57.885040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:58.023616 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.023591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2"] Jul 29 19:11:58.025229 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:11:58.025204 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e99e69d_cf24_4ff8_ba43_caeb3bb5f70a.slice/crio-1d5dd15e7d07eb95465fe6b05b54d337ca05928b1698b9cac4c905102caabdc3 WatchSource:0}: Error finding container 1d5dd15e7d07eb95465fe6b05b54d337ca05928b1698b9cac4c905102caabdc3: Status 404 returned error can't find the container with id 1d5dd15e7d07eb95465fe6b05b54d337ca05928b1698b9cac4c905102caabdc3 Jul 29 19:11:58.364105 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.364013 2575 generic.go:356] "Generic (PLEG): container finished" podID="1dee8e26-724b-4dc7-b4dd-a564b7f165ce" containerID="610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4" exitCode=0 Jul 29 19:11:58.364105 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.364058 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" Jul 29 19:11:58.364105 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.364089 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" event={"ID":"1dee8e26-724b-4dc7-b4dd-a564b7f165ce","Type":"ContainerDied","Data":"610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4"} Jul 29 19:11:58.364618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.364147 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb" event={"ID":"1dee8e26-724b-4dc7-b4dd-a564b7f165ce","Type":"ContainerDied","Data":"a8ae3aeffed583a275a767f59ec594f1a6a916970cb33f4fbd2dc16e1fdb1e5d"} Jul 29 19:11:58.364618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.364170 2575 scope.go:117] "RemoveContainer" containerID="610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4" Jul 29 19:11:58.366183 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.366160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" event={"ID":"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a","Type":"ContainerStarted","Data":"8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784"} Jul 29 19:11:58.366285 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.366199 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" event={"ID":"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a","Type":"ContainerStarted","Data":"1d5dd15e7d07eb95465fe6b05b54d337ca05928b1698b9cac4c905102caabdc3"} Jul 29 19:11:58.366285 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.366229 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:11:58.378435 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.378342 2575 scope.go:117] "RemoveContainer" containerID="610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4" Jul 29 19:11:58.378679 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:11:58.378656 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4\": container with ID starting with 610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4 not found: ID does not exist" containerID="610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4" Jul 29 19:11:58.378731 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.378688 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4"} err="failed to get container status \"610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4\": rpc error: code = NotFound desc = could not find container \"610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4\": container with ID starting with 610cc0c8892af0933fc98eb6ed4c2128ec28544e94ae43b366ad0688800059e4 not found: ID does not exist" Jul 29 19:11:58.394786 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.394760 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb"] Jul 29 19:11:58.397442 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.397419 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-57wbb"] Jul 29 19:11:58.413741 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:58.413699 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" podStartSLOduration=1.413686103 podStartE2EDuration="1.413686103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:58.412862998 +0000 UTC m=+699.429884095" watchObservedRunningTime="2026-07-29 19:11:58.413686103 +0000 UTC m=+699.430707199" Jul 29 19:11:59.549400 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:11:59.549364 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dee8e26-724b-4dc7-b4dd-a564b7f165ce" path="/var/lib/kubelet/pods/1dee8e26-724b-4dc7-b4dd-a564b7f165ce/volumes" Jul 29 19:12:09.376343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.376312 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:12:09.430281 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.430249 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h"] Jul 29 19:12:09.430508 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.430486 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" podUID="37983937-af76-4800-a48b-3250ee6b954b" containerName="manager" containerID="cri-o://3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f" gracePeriod=10 Jul 29 19:12:09.712985 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.712954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:12:09.789107 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.789075 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37983937-af76-4800-a48b-3250ee6b954b-extensions-socket-volume\") pod \"37983937-af76-4800-a48b-3250ee6b954b\" (UID: \"37983937-af76-4800-a48b-3250ee6b954b\") " Jul 29 19:12:09.789282 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.789206 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slsv2\" (UniqueName: \"kubernetes.io/projected/37983937-af76-4800-a48b-3250ee6b954b-kube-api-access-slsv2\") pod \"37983937-af76-4800-a48b-3250ee6b954b\" (UID: \"37983937-af76-4800-a48b-3250ee6b954b\") " Jul 29 19:12:09.789442 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.789421 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37983937-af76-4800-a48b-3250ee6b954b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37983937-af76-4800-a48b-3250ee6b954b" (UID: "37983937-af76-4800-a48b-3250ee6b954b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:12:09.791120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.791093 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37983937-af76-4800-a48b-3250ee6b954b-kube-api-access-slsv2" (OuterVolumeSpecName: "kube-api-access-slsv2") pod "37983937-af76-4800-a48b-3250ee6b954b" (UID: "37983937-af76-4800-a48b-3250ee6b954b"). InnerVolumeSpecName "kube-api-access-slsv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:12:09.890073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.890038 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slsv2\" (UniqueName: \"kubernetes.io/projected/37983937-af76-4800-a48b-3250ee6b954b-kube-api-access-slsv2\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:12:09.890073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:09.890065 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37983937-af76-4800-a48b-3250ee6b954b-extensions-socket-volume\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:12:10.434757 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.434726 2575 generic.go:356] "Generic (PLEG): container finished" podID="37983937-af76-4800-a48b-3250ee6b954b" containerID="3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f" exitCode=0 Jul 29 19:12:10.435185 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.434769 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" Jul 29 19:12:10.435185 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.434818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" event={"ID":"37983937-af76-4800-a48b-3250ee6b954b","Type":"ContainerDied","Data":"3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f"} Jul 29 19:12:10.435185 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.434868 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h" event={"ID":"37983937-af76-4800-a48b-3250ee6b954b","Type":"ContainerDied","Data":"ae5592db928bc088c3d33e3a1ea347db0ea939828edbc9e468a561732dbd6201"} Jul 29 19:12:10.435185 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.434889 2575 scope.go:117] "RemoveContainer" containerID="3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f" Jul 29 19:12:10.444508 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.444494 2575 scope.go:117] "RemoveContainer" containerID="3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f" Jul 29 19:12:10.444773 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:12:10.444755 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f\": container with ID starting with 3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f not found: ID does not exist" containerID="3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f" Jul 29 19:12:10.444840 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.444785 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f"} err="failed to get container status \"3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f\": rpc error: code = NotFound desc = could not find container \"3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f\": container with ID starting with 3749c4235444ab8a8a6553eae50621bcbc0990f85bde9b15c53b317285f28c3f not found: ID does not exist" Jul 29 19:12:10.460395 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.460373 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h"] Jul 29 19:12:10.465380 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:10.465359 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-jb77h"] Jul 29 19:12:11.548474 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:11.548440 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37983937-af76-4800-a48b-3250ee6b954b" path="/var/lib/kubelet/pods/37983937-af76-4800-a48b-3250ee6b954b/volumes" Jul 29 19:12:13.654224 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.654190 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh"] Jul 29 19:12:13.654628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.654580 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dee8e26-724b-4dc7-b4dd-a564b7f165ce" containerName="manager" Jul 29 19:12:13.654628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.654593 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dee8e26-724b-4dc7-b4dd-a564b7f165ce" containerName="manager" Jul 29 19:12:13.654628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.654605 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37983937-af76-4800-a48b-3250ee6b954b" containerName="manager" Jul 29 19:12:13.654628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.654611 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="37983937-af76-4800-a48b-3250ee6b954b" containerName="manager" Jul 29 19:12:13.654771 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.654679 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="37983937-af76-4800-a48b-3250ee6b954b" containerName="manager" Jul 29 19:12:13.654771 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.654689 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dee8e26-724b-4dc7-b4dd-a564b7f165ce" containerName="manager" Jul 29 19:12:13.659464 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.659445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.661870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.661849 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bhsgz\"" Jul 29 19:12:13.668949 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.668923 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh"] Jul 29 19:12:13.721871 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.721844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.722028 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.721878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bgt\" (UniqueName: \"kubernetes.io/projected/2f48062d-30b4-4a4b-ac41-3025ba43aad9-kube-api-access-r2bgt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.722028 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.721898 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.722028 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.721915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.722028 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.721998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.722212 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.722032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.722212 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.722076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.722212 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.722114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.722212 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.722135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822572 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822572 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bgt\" (UniqueName: \"kubernetes.io/projected/2f48062d-30b4-4a4b-ac41-3025ba43aad9-kube-api-access-r2bgt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.822822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.822748 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.823169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.823038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.823169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.823078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.823169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.823112 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.823314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.823235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.823406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.823391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.825013 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.824989 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.825241 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.825223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.830117 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.830092 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f48062d-30b4-4a4b-ac41-3025ba43aad9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.830210 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.830193 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bgt\" (UniqueName: \"kubernetes.io/projected/2f48062d-30b4-4a4b-ac41-3025ba43aad9-kube-api-access-r2bgt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fdsmh\" (UID: \"2f48062d-30b4-4a4b-ac41-3025ba43aad9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:13.970169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:13.970091 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:14.123742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:14.123708 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh"] Jul 29 19:12:14.126769 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:12:14.126741 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f48062d_30b4_4a4b_ac41_3025ba43aad9.slice/crio-f73a834a2b86f2361427a6d2de66bdd49a1cd811bce52d00559d555b9badb7a7 WatchSource:0}: Error finding container f73a834a2b86f2361427a6d2de66bdd49a1cd811bce52d00559d555b9badb7a7: Status 404 returned error can't find the container with id f73a834a2b86f2361427a6d2de66bdd49a1cd811bce52d00559d555b9badb7a7 Jul 29 19:12:14.135606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:14.129217 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:12:14.135606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:14.129382 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:12:14.135606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:14.129470 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:12:14.459663 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:14.459621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" event={"ID":"2f48062d-30b4-4a4b-ac41-3025ba43aad9","Type":"ContainerStarted","Data":"a980ff8509c81243570053112b8692cc17c4ee6a6887c56044794f7c611bf576"} Jul 29 19:12:14.459847 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:14.459715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" event={"ID":"2f48062d-30b4-4a4b-ac41-3025ba43aad9","Type":"ContainerStarted","Data":"f73a834a2b86f2361427a6d2de66bdd49a1cd811bce52d00559d555b9badb7a7"} Jul 29 19:12:14.483051 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:14.483005 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" podStartSLOduration=1.482989753 podStartE2EDuration="1.482989753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:12:14.480832453 +0000 UTC m=+715.497853549" watchObservedRunningTime="2026-07-29 19:12:14.482989753 +0000 UTC m=+715.500010849" Jul 29 19:12:14.971175 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:14.971136 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:15.976376 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:15.976346 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:16.472146 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:16.472112 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:16.473233 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:16.473211 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:12:29.397049 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.397010 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wwcvf"] Jul 29 19:12:29.460298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.460256 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wwcvf"] Jul 29 19:12:29.460455 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.460363 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:29.464352 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.464326 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 19:12:29.502970 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.502939 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wwcvf"] Jul 29 19:12:29.562122 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.562092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35248a17-7659-443c-9ce3-5805af12d399-config-file\") pod \"limitador-limitador-64c94fd64f-wwcvf\" (UID: \"35248a17-7659-443c-9ce3-5805af12d399\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:29.562299 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.562133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxvh\" (UniqueName: \"kubernetes.io/projected/35248a17-7659-443c-9ce3-5805af12d399-kube-api-access-pqxvh\") pod \"limitador-limitador-64c94fd64f-wwcvf\" (UID: \"35248a17-7659-443c-9ce3-5805af12d399\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:29.662677 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.662585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35248a17-7659-443c-9ce3-5805af12d399-config-file\") pod \"limitador-limitador-64c94fd64f-wwcvf\" (UID: \"35248a17-7659-443c-9ce3-5805af12d399\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:29.662677 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.662635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxvh\" (UniqueName: \"kubernetes.io/projected/35248a17-7659-443c-9ce3-5805af12d399-kube-api-access-pqxvh\") pod \"limitador-limitador-64c94fd64f-wwcvf\" (UID: \"35248a17-7659-443c-9ce3-5805af12d399\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:29.663179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.663160 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35248a17-7659-443c-9ce3-5805af12d399-config-file\") pod \"limitador-limitador-64c94fd64f-wwcvf\" (UID: \"35248a17-7659-443c-9ce3-5805af12d399\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:29.672775 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.672753 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxvh\" (UniqueName: \"kubernetes.io/projected/35248a17-7659-443c-9ce3-5805af12d399-kube-api-access-pqxvh\") pod \"limitador-limitador-64c94fd64f-wwcvf\" (UID: \"35248a17-7659-443c-9ce3-5805af12d399\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:29.772815 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.772772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:29.909166 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:29.909142 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wwcvf"] Jul 29 19:12:29.911385 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:12:29.911359 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35248a17_7659_443c_9ce3_5805af12d399.slice/crio-857f04093ce1544bd345405e4fe422fa381184e76bcf47adcfce9350a5f072ea WatchSource:0}: Error finding container 857f04093ce1544bd345405e4fe422fa381184e76bcf47adcfce9350a5f072ea: Status 404 returned error can't find the container with id 857f04093ce1544bd345405e4fe422fa381184e76bcf47adcfce9350a5f072ea Jul 29 19:12:30.527844 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.527811 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxdcq"] Jul 29 19:12:30.531612 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.531588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lxdcq" Jul 29 19:12:30.534696 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.534673 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qllfn\"" Jul 29 19:12:30.540935 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.540913 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxdcq"] Jul 29 19:12:30.548468 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.548437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" event={"ID":"35248a17-7659-443c-9ce3-5805af12d399","Type":"ContainerStarted","Data":"857f04093ce1544bd345405e4fe422fa381184e76bcf47adcfce9350a5f072ea"} Jul 29 19:12:30.672499 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.672459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xzds\" (UniqueName: \"kubernetes.io/projected/75f9be67-b73c-420e-ac52-47726c8aaa41-kube-api-access-9xzds\") pod \"authorino-75d556bd77-lxdcq\" (UID: \"75f9be67-b73c-420e-ac52-47726c8aaa41\") " pod="kuadrant-system/authorino-75d556bd77-lxdcq" Jul 29 19:12:30.774018 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.773855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xzds\" (UniqueName: \"kubernetes.io/projected/75f9be67-b73c-420e-ac52-47726c8aaa41-kube-api-access-9xzds\") pod \"authorino-75d556bd77-lxdcq\" (UID: \"75f9be67-b73c-420e-ac52-47726c8aaa41\") " pod="kuadrant-system/authorino-75d556bd77-lxdcq" Jul 29 19:12:30.784445 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.784374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xzds\" (UniqueName: \"kubernetes.io/projected/75f9be67-b73c-420e-ac52-47726c8aaa41-kube-api-access-9xzds\") pod \"authorino-75d556bd77-lxdcq\" (UID: \"75f9be67-b73c-420e-ac52-47726c8aaa41\") " pod="kuadrant-system/authorino-75d556bd77-lxdcq" Jul 29 19:12:30.841754 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:30.841717 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lxdcq" Jul 29 19:12:31.010375 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:31.010344 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxdcq"] Jul 29 19:12:31.013106 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:12:31.012681 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75f9be67_b73c_420e_ac52_47726c8aaa41.slice/crio-2225ab2b05b3ba61f4fe438ae47c5eeab345d672b8876a4807cd80ca1f06c874 WatchSource:0}: Error finding container 2225ab2b05b3ba61f4fe438ae47c5eeab345d672b8876a4807cd80ca1f06c874: Status 404 returned error can't find the container with id 2225ab2b05b3ba61f4fe438ae47c5eeab345d672b8876a4807cd80ca1f06c874 Jul 29 19:12:31.555226 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:31.555189 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lxdcq" event={"ID":"75f9be67-b73c-420e-ac52-47726c8aaa41","Type":"ContainerStarted","Data":"2225ab2b05b3ba61f4fe438ae47c5eeab345d672b8876a4807cd80ca1f06c874"} Jul 29 19:12:33.570184 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:33.570147 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" event={"ID":"35248a17-7659-443c-9ce3-5805af12d399","Type":"ContainerStarted","Data":"a72a10ef4a41246d7c23848d69c76bc9a99376e9990ab8672d19dc44d89cd699"} Jul 29 19:12:33.570683 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:33.570296 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:33.587845 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:33.587698 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" podStartSLOduration=1.792043753 podStartE2EDuration="4.587680944s" totalImagesPullingTime="2.795637191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:12:33.586369957 +0000 UTC m=+734.603391054" watchObservedRunningTime="2026-07-29 19:12:33.587680944 +0000 UTC m=+734.604702039" Jul 29 19:12:35.582094 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:35.582060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lxdcq" event={"ID":"75f9be67-b73c-420e-ac52-47726c8aaa41","Type":"ContainerStarted","Data":"ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc"} Jul 29 19:12:35.601795 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:35.601749 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-lxdcq" podStartSLOduration=2.012777026 podStartE2EDuration="5.601734078s" totalImagesPullingTime="3.588957052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:12:35.600141885 +0000 UTC m=+736.617162979" watchObservedRunningTime="2026-07-29 19:12:35.601734078 +0000 UTC m=+736.618755173" Jul 29 19:12:44.205876 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.205834 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wwcvf"] Jul 29 19:12:44.206326 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.206090 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" podUID="35248a17-7659-443c-9ce3-5805af12d399" containerName="limitador" containerID="cri-o://a72a10ef4a41246d7c23848d69c76bc9a99376e9990ab8672d19dc44d89cd699" gracePeriod=30 Jul 29 19:12:44.206841 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.206790 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:44.630000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.629972 2575 generic.go:356] "Generic (PLEG): container finished" podID="35248a17-7659-443c-9ce3-5805af12d399" containerID="a72a10ef4a41246d7c23848d69c76bc9a99376e9990ab8672d19dc44d89cd699" exitCode=0 Jul 29 19:12:44.630119 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.630038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" event={"ID":"35248a17-7659-443c-9ce3-5805af12d399","Type":"ContainerDied","Data":"a72a10ef4a41246d7c23848d69c76bc9a99376e9990ab8672d19dc44d89cd699"} Jul 29 19:12:44.751864 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.751840 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:44.909284 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.909246 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35248a17-7659-443c-9ce3-5805af12d399-config-file\") pod \"35248a17-7659-443c-9ce3-5805af12d399\" (UID: \"35248a17-7659-443c-9ce3-5805af12d399\") " Jul 29 19:12:44.909474 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.909326 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqxvh\" (UniqueName: \"kubernetes.io/projected/35248a17-7659-443c-9ce3-5805af12d399-kube-api-access-pqxvh\") pod \"35248a17-7659-443c-9ce3-5805af12d399\" (UID: \"35248a17-7659-443c-9ce3-5805af12d399\") " Jul 29 19:12:44.909736 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.909711 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35248a17-7659-443c-9ce3-5805af12d399-config-file" (OuterVolumeSpecName: "config-file") pod "35248a17-7659-443c-9ce3-5805af12d399" (UID: "35248a17-7659-443c-9ce3-5805af12d399"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:12:44.911397 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:44.911379 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35248a17-7659-443c-9ce3-5805af12d399-kube-api-access-pqxvh" (OuterVolumeSpecName: "kube-api-access-pqxvh") pod "35248a17-7659-443c-9ce3-5805af12d399" (UID: "35248a17-7659-443c-9ce3-5805af12d399"). InnerVolumeSpecName "kube-api-access-pqxvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:12:45.010847 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:45.010810 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35248a17-7659-443c-9ce3-5805af12d399-config-file\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:12:45.010847 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:45.010840 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqxvh\" (UniqueName: \"kubernetes.io/projected/35248a17-7659-443c-9ce3-5805af12d399-kube-api-access-pqxvh\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:12:45.640827 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:45.640800 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" Jul 29 19:12:45.641247 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:45.640836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wwcvf" event={"ID":"35248a17-7659-443c-9ce3-5805af12d399","Type":"ContainerDied","Data":"857f04093ce1544bd345405e4fe422fa381184e76bcf47adcfce9350a5f072ea"} Jul 29 19:12:45.641247 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:45.640926 2575 scope.go:117] "RemoveContainer" containerID="a72a10ef4a41246d7c23848d69c76bc9a99376e9990ab8672d19dc44d89cd699" Jul 29 19:12:45.657064 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:45.657039 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wwcvf"] Jul 29 19:12:45.661878 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:45.661859 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wwcvf"] Jul 29 19:12:47.384865 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.384830 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4"] Jul 29 19:12:47.385473 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.385456 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35248a17-7659-443c-9ce3-5805af12d399" containerName="limitador" Jul 29 19:12:47.385546 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.385476 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="35248a17-7659-443c-9ce3-5805af12d399" containerName="limitador" Jul 29 19:12:47.385595 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.385583 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="35248a17-7659-443c-9ce3-5805af12d399" containerName="limitador" Jul 29 19:12:47.389382 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.389356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:47.392001 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.391981 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-r88xw\"" Jul 29 19:12:47.392100 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.391998 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 19:12:47.392960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.392947 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 19:12:47.393014 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.392960 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 19:12:47.397653 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.397633 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4"] Jul 29 19:12:47.531215 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.531178 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f449a28f-feef-4724-82d8-cb9ec4d19279-data\") pod \"postgres-69fb4594cf-jc9w4\" (UID: \"f449a28f-feef-4724-82d8-cb9ec4d19279\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:47.531380 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.531231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8xx8\" (UniqueName: \"kubernetes.io/projected/f449a28f-feef-4724-82d8-cb9ec4d19279-kube-api-access-w8xx8\") pod \"postgres-69fb4594cf-jc9w4\" (UID: \"f449a28f-feef-4724-82d8-cb9ec4d19279\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:47.559707 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.559678 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35248a17-7659-443c-9ce3-5805af12d399" path="/var/lib/kubelet/pods/35248a17-7659-443c-9ce3-5805af12d399/volumes" Jul 29 19:12:47.631856 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.631818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8xx8\" (UniqueName: \"kubernetes.io/projected/f449a28f-feef-4724-82d8-cb9ec4d19279-kube-api-access-w8xx8\") pod \"postgres-69fb4594cf-jc9w4\" (UID: \"f449a28f-feef-4724-82d8-cb9ec4d19279\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:47.632178 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.631968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f449a28f-feef-4724-82d8-cb9ec4d19279-data\") pod \"postgres-69fb4594cf-jc9w4\" (UID: \"f449a28f-feef-4724-82d8-cb9ec4d19279\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:47.632363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.632341 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f449a28f-feef-4724-82d8-cb9ec4d19279-data\") pod \"postgres-69fb4594cf-jc9w4\" (UID: \"f449a28f-feef-4724-82d8-cb9ec4d19279\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:47.646995 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.646929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8xx8\" (UniqueName: \"kubernetes.io/projected/f449a28f-feef-4724-82d8-cb9ec4d19279-kube-api-access-w8xx8\") pod \"postgres-69fb4594cf-jc9w4\" (UID: \"f449a28f-feef-4724-82d8-cb9ec4d19279\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:47.699944 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.699915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:47.832212 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:47.832179 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4"] Jul 29 19:12:47.834326 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:12:47.834299 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf449a28f_feef_4724_82d8_cb9ec4d19279.slice/crio-ef49194002d85a5c85794cbd560ebc4cdf48c94e8ca24db262ac74c386a369b1 WatchSource:0}: Error finding container ef49194002d85a5c85794cbd560ebc4cdf48c94e8ca24db262ac74c386a369b1: Status 404 returned error can't find the container with id ef49194002d85a5c85794cbd560ebc4cdf48c94e8ca24db262ac74c386a369b1 Jul 29 19:12:48.658654 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:48.658611 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" event={"ID":"f449a28f-feef-4724-82d8-cb9ec4d19279","Type":"ContainerStarted","Data":"ef49194002d85a5c85794cbd560ebc4cdf48c94e8ca24db262ac74c386a369b1"} Jul 29 19:12:53.693887 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:53.693793 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" event={"ID":"f449a28f-feef-4724-82d8-cb9ec4d19279","Type":"ContainerStarted","Data":"d0bca1d0e4e27dcad0953651b190d1cd6471c41a31c5b33f62a419a1f120e74a"} Jul 29 19:12:53.693887 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:53.693862 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:12:53.715052 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:53.715002 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" podStartSLOduration=1.146069722 podStartE2EDuration="6.714989212s" totalImagesPullingTime="5.56891949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:12:53.712092382 +0000 UTC m=+754.729113479" watchObservedRunningTime="2026-07-29 19:12:53.714989212 +0000 UTC m=+754.732010306" Jul 29 19:12:59.727279 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:12:59.727248 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jc9w4" Jul 29 19:13:00.457587 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.457549 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-45phz"] Jul 29 19:13:00.462093 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.462076 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.464722 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.464697 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 19:13:00.468068 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.468045 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-45phz"] Jul 29 19:13:00.514993 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.514961 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-45phz"] Jul 29 19:13:00.515528 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:13:00.515483 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pbp8z tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-pbp8z tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-45phz" podUID="eb917c5f-1bc8-4b7b-bff8-fa454fb8d901" Jul 29 19:13:00.558308 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.558279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbp8z\" (UniqueName: \"kubernetes.io/projected/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-kube-api-access-pbp8z\") pod \"authorino-76877994-45phz\" (UID: \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\") " pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.558446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.558324 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-tls-cert\") pod \"authorino-76877994-45phz\" (UID: \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\") " pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.659509 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.659467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbp8z\" (UniqueName: \"kubernetes.io/projected/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-kube-api-access-pbp8z\") pod \"authorino-76877994-45phz\" (UID: \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\") " pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.659509 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.659534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-tls-cert\") pod \"authorino-76877994-45phz\" (UID: \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\") " pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.661997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.661972 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-tls-cert\") pod \"authorino-76877994-45phz\" (UID: \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\") " pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.668115 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.668096 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbp8z\" (UniqueName: \"kubernetes.io/projected/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-kube-api-access-pbp8z\") pod \"authorino-76877994-45phz\" (UID: \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\") " pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.731489 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.731423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.738306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.738283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:00.861259 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.861215 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-tls-cert\") pod \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\" (UID: \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\") " Jul 29 19:13:00.861259 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.861273 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbp8z\" (UniqueName: \"kubernetes.io/projected/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-kube-api-access-pbp8z\") pod \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\" (UID: \"eb917c5f-1bc8-4b7b-bff8-fa454fb8d901\") " Jul 29 19:13:00.863269 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.863242 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "eb917c5f-1bc8-4b7b-bff8-fa454fb8d901" (UID: "eb917c5f-1bc8-4b7b-bff8-fa454fb8d901"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:13:00.863390 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.863370 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-kube-api-access-pbp8z" (OuterVolumeSpecName: "kube-api-access-pbp8z") pod "eb917c5f-1bc8-4b7b-bff8-fa454fb8d901" (UID: "eb917c5f-1bc8-4b7b-bff8-fa454fb8d901"). InnerVolumeSpecName "kube-api-access-pbp8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:13:00.962374 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.962341 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-tls-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:13:00.962374 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:00.962368 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbp8z\" (UniqueName: \"kubernetes.io/projected/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901-kube-api-access-pbp8z\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:13:01.736583 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:01.736550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-45phz" Jul 29 19:13:01.767879 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:01.767848 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-45phz"] Jul 29 19:13:01.771640 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:01.771615 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-45phz"] Jul 29 19:13:02.309370 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.309326 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxdcq"] Jul 29 19:13:02.309594 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.309567 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-lxdcq" podUID="75f9be67-b73c-420e-ac52-47726c8aaa41" containerName="authorino" containerID="cri-o://ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc" gracePeriod=30 Jul 29 19:13:02.556101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.556076 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lxdcq" Jul 29 19:13:02.678302 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.678275 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xzds\" (UniqueName: \"kubernetes.io/projected/75f9be67-b73c-420e-ac52-47726c8aaa41-kube-api-access-9xzds\") pod \"75f9be67-b73c-420e-ac52-47726c8aaa41\" (UID: \"75f9be67-b73c-420e-ac52-47726c8aaa41\") " Jul 29 19:13:02.680701 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.680663 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f9be67-b73c-420e-ac52-47726c8aaa41-kube-api-access-9xzds" (OuterVolumeSpecName: "kube-api-access-9xzds") pod "75f9be67-b73c-420e-ac52-47726c8aaa41" (UID: "75f9be67-b73c-420e-ac52-47726c8aaa41"). InnerVolumeSpecName "kube-api-access-9xzds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:13:02.742788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.742760 2575 generic.go:356] "Generic (PLEG): container finished" podID="75f9be67-b73c-420e-ac52-47726c8aaa41" containerID="ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc" exitCode=0 Jul 29 19:13:02.743118 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.742806 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lxdcq" Jul 29 19:13:02.743118 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.742842 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lxdcq" event={"ID":"75f9be67-b73c-420e-ac52-47726c8aaa41","Type":"ContainerDied","Data":"ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc"} Jul 29 19:13:02.743118 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.742884 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lxdcq" event={"ID":"75f9be67-b73c-420e-ac52-47726c8aaa41","Type":"ContainerDied","Data":"2225ab2b05b3ba61f4fe438ae47c5eeab345d672b8876a4807cd80ca1f06c874"} Jul 29 19:13:02.743118 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.742902 2575 scope.go:117] "RemoveContainer" containerID="ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc" Jul 29 19:13:02.751796 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.751778 2575 scope.go:117] "RemoveContainer" containerID="ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc" Jul 29 19:13:02.752046 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:13:02.752028 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc\": container with ID starting with ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc not found: ID does not exist" containerID="ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc" Jul 29 19:13:02.752113 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.752055 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc"} err="failed to get container status \"ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc\": rpc error: code = NotFound desc = could not find container \"ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc\": container with ID starting with ffcb8e751a8569ed9811c9b694d8ece1c7c5f7df7f05b6b6a8436d0adf26b5dc not found: ID does not exist" Jul 29 19:13:02.764109 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.764083 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxdcq"] Jul 29 19:13:02.768163 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.768144 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxdcq"] Jul 29 19:13:02.779374 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:02.779353 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xzds\" (UniqueName: \"kubernetes.io/projected/75f9be67-b73c-420e-ac52-47726c8aaa41-kube-api-access-9xzds\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:13:03.549806 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:03.549773 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f9be67-b73c-420e-ac52-47726c8aaa41" path="/var/lib/kubelet/pods/75f9be67-b73c-420e-ac52-47726c8aaa41/volumes" Jul 29 19:13:03.550126 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:03.550113 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb917c5f-1bc8-4b7b-bff8-fa454fb8d901" path="/var/lib/kubelet/pods/eb917c5f-1bc8-4b7b-bff8-fa454fb8d901/volumes" Jul 29 19:13:06.949421 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:06.949386 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68cb4c8c66-2v7b9"] Jul 29 19:13:06.949843 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:06.949794 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f9be67-b73c-420e-ac52-47726c8aaa41" containerName="authorino" Jul 29 19:13:06.949843 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:06.949805 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f9be67-b73c-420e-ac52-47726c8aaa41" containerName="authorino" Jul 29 19:13:06.949924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:06.949865 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f9be67-b73c-420e-ac52-47726c8aaa41" containerName="authorino" Jul 29 19:13:07.173360 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.173314 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68cb4c8c66-2v7b9"] Jul 29 19:13:07.173360 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.173366 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-vxv8r"] Jul 29 19:13:07.173645 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.173418 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:07.177152 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.177123 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zgvq9\"" Jul 29 19:13:07.177152 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.177138 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 19:13:07.177152 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.177150 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 19:13:07.194700 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.194672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-vxv8r"] Jul 29 19:13:07.194842 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.194763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:07.322222 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.322179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhth\" (UniqueName: \"kubernetes.io/projected/87e54949-6459-4e91-b2a9-146540aaed4e-kube-api-access-fkhth\") pod \"maas-controller-68cb4c8c66-2v7b9\" (UID: \"87e54949-6459-4e91-b2a9-146540aaed4e\") " pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:07.322396 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.322320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7lf\" (UniqueName: \"kubernetes.io/projected/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-kube-api-access-mb7lf\") pod \"maas-controller-7c6f6796c6-vxv8r\" (UID: \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\") " pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:07.322396 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.322347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87e54949-6459-4e91-b2a9-146540aaed4e-webhook-cert\") pod \"maas-controller-68cb4c8c66-2v7b9\" (UID: \"87e54949-6459-4e91-b2a9-146540aaed4e\") " pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:07.322396 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.322387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-webhook-cert\") pod \"maas-controller-7c6f6796c6-vxv8r\" (UID: \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\") " pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:07.423741 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.423700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-webhook-cert\") pod \"maas-controller-7c6f6796c6-vxv8r\" (UID: \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\") " pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:07.423905 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.423757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhth\" (UniqueName: \"kubernetes.io/projected/87e54949-6459-4e91-b2a9-146540aaed4e-kube-api-access-fkhth\") pod \"maas-controller-68cb4c8c66-2v7b9\" (UID: \"87e54949-6459-4e91-b2a9-146540aaed4e\") " pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:07.423905 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.423835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7lf\" (UniqueName: \"kubernetes.io/projected/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-kube-api-access-mb7lf\") pod \"maas-controller-7c6f6796c6-vxv8r\" (UID: \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\") " pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:07.423986 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.423959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87e54949-6459-4e91-b2a9-146540aaed4e-webhook-cert\") pod \"maas-controller-68cb4c8c66-2v7b9\" (UID: \"87e54949-6459-4e91-b2a9-146540aaed4e\") " pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:07.426199 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.426173 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-webhook-cert\") pod \"maas-controller-7c6f6796c6-vxv8r\" (UID: \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\") " pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:07.426321 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.426298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87e54949-6459-4e91-b2a9-146540aaed4e-webhook-cert\") pod \"maas-controller-68cb4c8c66-2v7b9\" (UID: \"87e54949-6459-4e91-b2a9-146540aaed4e\") " pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:07.432178 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.432153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhth\" (UniqueName: \"kubernetes.io/projected/87e54949-6459-4e91-b2a9-146540aaed4e-kube-api-access-fkhth\") pod \"maas-controller-68cb4c8c66-2v7b9\" (UID: \"87e54949-6459-4e91-b2a9-146540aaed4e\") " pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:07.432431 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.432409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7lf\" (UniqueName: \"kubernetes.io/projected/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-kube-api-access-mb7lf\") pod \"maas-controller-7c6f6796c6-vxv8r\" (UID: \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\") " pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:07.483325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.483239 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:07.504422 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.504390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:07.659787 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.659754 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68cb4c8c66-2v7b9"] Jul 29 19:13:07.663004 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:13:07.662976 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e54949_6459_4e91_b2a9_146540aaed4e.slice/crio-a83cfd64fabc695e123b02634db0e56ca40e24753825b1745ae7905f3ade2237 WatchSource:0}: Error finding container a83cfd64fabc695e123b02634db0e56ca40e24753825b1745ae7905f3ade2237: Status 404 returned error can't find the container with id a83cfd64fabc695e123b02634db0e56ca40e24753825b1745ae7905f3ade2237 Jul 29 19:13:07.673456 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.673431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-vxv8r"] Jul 29 19:13:07.675983 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:13:07.675958 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2a6b7f_e26a_4728_af0d_2163abe5c68f.slice/crio-2bbd69465e2e723e638075cfa72f8663b20f285a705843f78ca699934f08821a WatchSource:0}: Error finding container 2bbd69465e2e723e638075cfa72f8663b20f285a705843f78ca699934f08821a: Status 404 returned error can't find the container with id 2bbd69465e2e723e638075cfa72f8663b20f285a705843f78ca699934f08821a Jul 29 19:13:07.775100 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.775014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" event={"ID":"8d2a6b7f-e26a-4728-af0d-2163abe5c68f","Type":"ContainerStarted","Data":"2bbd69465e2e723e638075cfa72f8663b20f285a705843f78ca699934f08821a"} Jul 29 19:13:07.776050 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:07.776022 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" event={"ID":"87e54949-6459-4e91-b2a9-146540aaed4e","Type":"ContainerStarted","Data":"a83cfd64fabc695e123b02634db0e56ca40e24753825b1745ae7905f3ade2237"} Jul 29 19:13:10.800915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:10.800880 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" event={"ID":"87e54949-6459-4e91-b2a9-146540aaed4e","Type":"ContainerStarted","Data":"e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661"} Jul 29 19:13:10.801441 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:10.800950 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:10.802413 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:10.802393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" event={"ID":"8d2a6b7f-e26a-4728-af0d-2163abe5c68f","Type":"ContainerStarted","Data":"008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2"} Jul 29 19:13:10.802541 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:10.802530 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:10.820348 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:10.820303 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" podStartSLOduration=2.309143234 podStartE2EDuration="4.820286902s" totalImagesPullingTime="2.511143668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:10.818994387 +0000 UTC m=+771.836015481" watchObservedRunningTime="2026-07-29 19:13:10.820286902 +0000 UTC m=+771.837308034" Jul 29 19:13:10.842374 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:10.842320 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" podStartSLOduration=1.348438451 podStartE2EDuration="3.842304103s" totalImagesPullingTime="2.493865652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:10.840063466 +0000 UTC m=+771.857084560" watchObservedRunningTime="2026-07-29 19:13:10.842304103 +0000 UTC m=+771.859325282" Jul 29 19:13:21.810148 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:21.810118 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:21.810574 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:21.810187 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:13:21.881316 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:21.881270 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68cb4c8c66-2v7b9"] Jul 29 19:13:21.881510 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:21.881483 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" podUID="87e54949-6459-4e91-b2a9-146540aaed4e" containerName="manager" containerID="cri-o://e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661" gracePeriod=30 Jul 29 19:13:22.130860 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.130835 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:22.258585 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.258550 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkhth\" (UniqueName: \"kubernetes.io/projected/87e54949-6459-4e91-b2a9-146540aaed4e-kube-api-access-fkhth\") pod \"87e54949-6459-4e91-b2a9-146540aaed4e\" (UID: \"87e54949-6459-4e91-b2a9-146540aaed4e\") " Jul 29 19:13:22.258758 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.258643 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87e54949-6459-4e91-b2a9-146540aaed4e-webhook-cert\") pod \"87e54949-6459-4e91-b2a9-146540aaed4e\" (UID: \"87e54949-6459-4e91-b2a9-146540aaed4e\") " Jul 29 19:13:22.260789 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.260750 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e54949-6459-4e91-b2a9-146540aaed4e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "87e54949-6459-4e91-b2a9-146540aaed4e" (UID: "87e54949-6459-4e91-b2a9-146540aaed4e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:13:22.260903 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.260787 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e54949-6459-4e91-b2a9-146540aaed4e-kube-api-access-fkhth" (OuterVolumeSpecName: "kube-api-access-fkhth") pod "87e54949-6459-4e91-b2a9-146540aaed4e" (UID: "87e54949-6459-4e91-b2a9-146540aaed4e"). InnerVolumeSpecName "kube-api-access-fkhth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:13:22.360185 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.360149 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkhth\" (UniqueName: \"kubernetes.io/projected/87e54949-6459-4e91-b2a9-146540aaed4e-kube-api-access-fkhth\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:13:22.360185 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.360182 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87e54949-6459-4e91-b2a9-146540aaed4e-webhook-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:13:22.867911 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.867880 2575 generic.go:356] "Generic (PLEG): container finished" podID="87e54949-6459-4e91-b2a9-146540aaed4e" containerID="e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661" exitCode=0 Jul 29 19:13:22.868318 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.867921 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" Jul 29 19:13:22.868318 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.867940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" event={"ID":"87e54949-6459-4e91-b2a9-146540aaed4e","Type":"ContainerDied","Data":"e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661"} Jul 29 19:13:22.868318 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.867985 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68cb4c8c66-2v7b9" event={"ID":"87e54949-6459-4e91-b2a9-146540aaed4e","Type":"ContainerDied","Data":"a83cfd64fabc695e123b02634db0e56ca40e24753825b1745ae7905f3ade2237"} Jul 29 19:13:22.868318 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.868007 2575 scope.go:117] "RemoveContainer" containerID="e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661" Jul 29 19:13:22.877340 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.877322 2575 scope.go:117] "RemoveContainer" containerID="e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661" Jul 29 19:13:22.877600 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:13:22.877582 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661\": container with ID starting with e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661 not found: ID does not exist" containerID="e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661" Jul 29 19:13:22.877653 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.877609 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661"} err="failed to get container status \"e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661\": rpc error: code = NotFound desc = could not find container \"e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661\": container with ID starting with e5c287cb7d38c97b450fbc07b96e1b2d596039f73ff1dfcf2e753ad731a16661 not found: ID does not exist" Jul 29 19:13:22.889913 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.889889 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68cb4c8c66-2v7b9"] Jul 29 19:13:22.896970 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:22.896952 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68cb4c8c66-2v7b9"] Jul 29 19:13:23.549378 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:23.549348 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e54949-6459-4e91-b2a9-146540aaed4e" path="/var/lib/kubelet/pods/87e54949-6459-4e91-b2a9-146540aaed4e/volumes" Jul 29 19:13:43.492048 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.491964 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698"] Jul 29 19:13:43.492538 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.492382 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e54949-6459-4e91-b2a9-146540aaed4e" containerName="manager" Jul 29 19:13:43.492538 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.492393 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e54949-6459-4e91-b2a9-146540aaed4e" containerName="manager" Jul 29 19:13:43.492538 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.492452 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e54949-6459-4e91-b2a9-146540aaed4e" containerName="manager" Jul 29 19:13:43.498971 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.498950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.501946 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.501917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 29 19:13:43.503232 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.503206 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5bzmf\"" Jul 29 19:13:43.505080 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.505060 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698"] Jul 29 19:13:43.542282 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.542251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/723506d5-645a-48c8-96c2-4acd2bb8e126-maas-api-tls\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.542433 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.542322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwv52\" (UniqueName: \"kubernetes.io/projected/723506d5-645a-48c8-96c2-4acd2bb8e126-kube-api-access-hwv52\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.542433 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.542381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/723506d5-645a-48c8-96c2-4acd2bb8e126-openshift-service-ca\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.643642 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.643608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwv52\" (UniqueName: \"kubernetes.io/projected/723506d5-645a-48c8-96c2-4acd2bb8e126-kube-api-access-hwv52\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.643861 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.643689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/723506d5-645a-48c8-96c2-4acd2bb8e126-openshift-service-ca\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.643861 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.643759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/723506d5-645a-48c8-96c2-4acd2bb8e126-maas-api-tls\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.644336 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.644314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/723506d5-645a-48c8-96c2-4acd2bb8e126-openshift-service-ca\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.646125 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.646103 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/723506d5-645a-48c8-96c2-4acd2bb8e126-maas-api-tls\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.653111 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.653081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwv52\" (UniqueName: \"kubernetes.io/projected/723506d5-645a-48c8-96c2-4acd2bb8e126-kube-api-access-hwv52\") pod \"maas-api-85cdb98bf6-wv698\" (UID: \"723506d5-645a-48c8-96c2-4acd2bb8e126\") " pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.810400 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.810318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:43.966618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.966592 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698"] Jul 29 19:13:43.969405 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:13:43.969347 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723506d5_645a_48c8_96c2_4acd2bb8e126.slice/crio-d4944bc848a66e27a4a84e75375934dc0dab2825f355cf0207ef25a5ad59b7ad WatchSource:0}: Error finding container d4944bc848a66e27a4a84e75375934dc0dab2825f355cf0207ef25a5ad59b7ad: Status 404 returned error can't find the container with id d4944bc848a66e27a4a84e75375934dc0dab2825f355cf0207ef25a5ad59b7ad Jul 29 19:13:43.984478 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:43.984442 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" event={"ID":"723506d5-645a-48c8-96c2-4acd2bb8e126","Type":"ContainerStarted","Data":"d4944bc848a66e27a4a84e75375934dc0dab2825f355cf0207ef25a5ad59b7ad"} Jul 29 19:13:44.127347 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.127314 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb"] Jul 29 19:13:44.132079 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.132058 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:44.134689 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.134664 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 29 19:13:44.134833 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.134813 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gp7r8\"" Jul 29 19:13:44.143575 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.143505 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb"] Jul 29 19:13:44.250656 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.250613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxf9k\" (UniqueName: \"kubernetes.io/projected/265893ee-6692-4d22-9d62-b4749ebc6b0b-kube-api-access-jxf9k\") pod \"payload-pre-processing-687dbb6cf6-2vdwb\" (UID: \"265893ee-6692-4d22-9d62-b4749ebc6b0b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:44.250836 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.250681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265893ee-6692-4d22-9d62-b4749ebc6b0b-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2vdwb\" (UID: \"265893ee-6692-4d22-9d62-b4749ebc6b0b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:44.351181 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.351142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxf9k\" (UniqueName: \"kubernetes.io/projected/265893ee-6692-4d22-9d62-b4749ebc6b0b-kube-api-access-jxf9k\") pod \"payload-pre-processing-687dbb6cf6-2vdwb\" (UID: \"265893ee-6692-4d22-9d62-b4749ebc6b0b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:44.351358 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.351201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265893ee-6692-4d22-9d62-b4749ebc6b0b-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2vdwb\" (UID: \"265893ee-6692-4d22-9d62-b4749ebc6b0b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:44.352017 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.351988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265893ee-6692-4d22-9d62-b4749ebc6b0b-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2vdwb\" (UID: \"265893ee-6692-4d22-9d62-b4749ebc6b0b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:44.362122 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.362093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxf9k\" (UniqueName: \"kubernetes.io/projected/265893ee-6692-4d22-9d62-b4749ebc6b0b-kube-api-access-jxf9k\") pod \"payload-pre-processing-687dbb6cf6-2vdwb\" (UID: \"265893ee-6692-4d22-9d62-b4749ebc6b0b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:44.446011 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.445914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:44.592376 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.592344 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb"] Jul 29 19:13:44.595086 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:13:44.595053 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265893ee_6692_4d22_9d62_b4749ebc6b0b.slice/crio-12ef8d7696b897807a189d33964237a9e1979c188e082b65e431875b77a045ae WatchSource:0}: Error finding container 12ef8d7696b897807a189d33964237a9e1979c188e082b65e431875b77a045ae: Status 404 returned error can't find the container with id 12ef8d7696b897807a189d33964237a9e1979c188e082b65e431875b77a045ae Jul 29 19:13:44.994349 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:44.994310 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" event={"ID":"265893ee-6692-4d22-9d62-b4749ebc6b0b","Type":"ContainerStarted","Data":"12ef8d7696b897807a189d33964237a9e1979c188e082b65e431875b77a045ae"} Jul 29 19:13:46.005363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:46.005326 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" event={"ID":"723506d5-645a-48c8-96c2-4acd2bb8e126","Type":"ContainerStarted","Data":"064fc6c64bde9a9e9180bcb7504f76c90fd831fbbf22fd8527c6552e5c257e36"} Jul 29 19:13:46.005817 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:46.005606 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:46.023760 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:46.023700 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" podStartSLOduration=1.188026493 podStartE2EDuration="3.023675237s" totalImagesPullingTime="1.835648744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:46.021953168 +0000 UTC m=+807.038974267" watchObservedRunningTime="2026-07-29 19:13:46.023675237 +0000 UTC m=+807.040696338" Jul 29 19:13:48.021133 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:48.021098 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" event={"ID":"265893ee-6692-4d22-9d62-b4749ebc6b0b","Type":"ContainerStarted","Data":"f3f97bc815e25ec44a23c35f7cc56908c2c2ebc2ab66027df567bbe8e7f27882"} Jul 29 19:13:48.021508 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:48.021267 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:13:48.041817 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:48.041752 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" podStartSLOduration=1.261879741 podStartE2EDuration="4.041731998s" totalImagesPullingTime="2.779852257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:48.03807248 +0000 UTC m=+809.055093574" watchObservedRunningTime="2026-07-29 19:13:48.041731998 +0000 UTC m=+809.058753095" Jul 29 19:13:52.020424 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:52.020347 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85cdb98bf6-wv698" Jul 29 19:13:52.046436 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:52.046401 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f48062d-30b4-4a4b-ac41-3025ba43aad9" containerID="a980ff8509c81243570053112b8692cc17c4ee6a6887c56044794f7c611bf576" exitCode=0 Jul 29 19:13:52.046662 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:52.046457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" event={"ID":"2f48062d-30b4-4a4b-ac41-3025ba43aad9","Type":"ContainerDied","Data":"a980ff8509c81243570053112b8692cc17c4ee6a6887c56044794f7c611bf576"} Jul 29 19:13:52.046972 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:52.046954 2575 scope.go:117] "RemoveContainer" containerID="a980ff8509c81243570053112b8692cc17c4ee6a6887c56044794f7c611bf576" Jul 29 19:13:52.048156 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:52.048092 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:13:52.048233 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:52.048213 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:13:52.048278 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:52.048263 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:13:53.054646 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:53.054611 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" event={"ID":"2f48062d-30b4-4a4b-ac41-3025ba43aad9","Type":"ContainerStarted","Data":"ca3f15587b73aa04c6f899b5515d07d2d4331fe1e2d1b6f3fc4fe5a1eb1cc12b"} Jul 29 19:13:53.971228 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:53.971186 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:13:53.971446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:53.971373 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fdsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 29 19:13:53.971446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:53.971415 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" podUID="2f48062d-30b4-4a4b-ac41-3025ba43aad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 29 19:13:54.971374 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:54.971331 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fdsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 29 19:13:54.971822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:54.971392 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" podUID="2f48062d-30b4-4a4b-ac41-3025ba43aad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 29 19:13:55.971443 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:55.971402 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fdsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 29 19:13:55.971937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:55.971468 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" podUID="2f48062d-30b4-4a4b-ac41-3025ba43aad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 29 19:13:56.970895 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:56.970855 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fdsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 29 19:13:56.971071 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:56.970915 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" podUID="2f48062d-30b4-4a4b-ac41-3025ba43aad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 29 19:13:57.974900 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:57.974866 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:13:57.975314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:57.975265 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:13:57.975847 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:57.975828 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fdsmh" Jul 29 19:13:59.029731 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:13:59.029700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2vdwb" Jul 29 19:14:04.783389 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.783352 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd"] Jul 29 19:14:04.786255 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.786237 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.788923 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.788903 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 19:14:04.789029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.788933 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 19:14:04.789894 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.789878 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lj9bm\"" Jul 29 19:14:04.789984 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.789920 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 29 19:14:04.797601 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.797579 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd"] Jul 29 19:14:04.849803 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.849774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.849969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.849833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c290c0c-a853-48c5-bf92-c5b0b01583af-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.849969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.849873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.849969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.849913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.849969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.849941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzhz8\" (UniqueName: \"kubernetes.io/projected/3c290c0c-a853-48c5-bf92-c5b0b01583af-kube-api-access-vzhz8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.850205 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.850070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951216 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c290c0c-a853-48c5-bf92-c5b0b01583af-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzhz8\" (UniqueName: \"kubernetes.io/projected/3c290c0c-a853-48c5-bf92-c5b0b01583af-kube-api-access-vzhz8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951716 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951893 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.951893 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.951741 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.954046 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.954014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c290c0c-a853-48c5-bf92-c5b0b01583af-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.954161 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.954128 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c290c0c-a853-48c5-bf92-c5b0b01583af-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:04.960800 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:04.960720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzhz8\" (UniqueName: \"kubernetes.io/projected/3c290c0c-a853-48c5-bf92-c5b0b01583af-kube-api-access-vzhz8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd\" (UID: \"3c290c0c-a853-48c5-bf92-c5b0b01583af\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:05.097438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:05.097354 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:05.240362 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:05.240333 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd"] Jul 29 19:14:05.242433 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:14:05.242399 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c290c0c_a853_48c5_bf92_c5b0b01583af.slice/crio-4688d30ba2cbb5956bb87c1b9448b19e32e4866bad89aa5b4425cc45b5bc6f05 WatchSource:0}: Error finding container 4688d30ba2cbb5956bb87c1b9448b19e32e4866bad89aa5b4425cc45b5bc6f05: Status 404 returned error can't find the container with id 4688d30ba2cbb5956bb87c1b9448b19e32e4866bad89aa5b4425cc45b5bc6f05 Jul 29 19:14:06.132165 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:06.132124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" event={"ID":"3c290c0c-a853-48c5-bf92-c5b0b01583af","Type":"ContainerStarted","Data":"4688d30ba2cbb5956bb87c1b9448b19e32e4866bad89aa5b4425cc45b5bc6f05"} Jul 29 19:14:07.139371 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:07.139328 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" event={"ID":"3c290c0c-a853-48c5-bf92-c5b0b01583af","Type":"ContainerStarted","Data":"79ffdcc9d32d32633c6de548b7a07c512b6ea4e65a79d48847e8afe6a8784507"} Jul 29 19:14:07.163709 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:07.163644 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" podStartSLOduration=1.934193461 podStartE2EDuration="3.163623039s" totalImagesPullingTime="1.229429578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:14:07.160888841 +0000 UTC m=+828.177909938" watchObservedRunningTime="2026-07-29 19:14:07.163623039 +0000 UTC m=+828.180644135" Jul 29 19:14:15.098011 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:15.097968 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:15.098500 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:15.098030 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:15.110590 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:15.110563 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:14:15.194666 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:14:15.194637 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd" Jul 29 19:15:19.402664 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.402580 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl"] Jul 29 19:15:19.406409 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.406384 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.409153 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.409124 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 29 19:15:19.419031 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.418994 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl"] Jul 29 19:15:19.427639 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.427609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.427923 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.427900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdcf\" (UniqueName: \"kubernetes.io/projected/f84d1fa4-d0a3-4794-8e67-8f028d00724e-kube-api-access-8hdcf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.428060 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.428043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.428170 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.428156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.428278 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.428265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.428407 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.428393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f84d1fa4-d0a3-4794-8e67-8f028d00724e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.529298 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.529264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdcf\" (UniqueName: \"kubernetes.io/projected/f84d1fa4-d0a3-4794-8e67-8f028d00724e-kube-api-access-8hdcf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.529487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.529324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.529487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.529353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.529487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.529387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.529487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.529438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f84d1fa4-d0a3-4794-8e67-8f028d00724e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.530098 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.529537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.530098 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.529842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.530231 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.530204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.530497 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.530463 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.531835 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.531811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f84d1fa4-d0a3-4794-8e67-8f028d00724e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.532592 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.532569 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f84d1fa4-d0a3-4794-8e67-8f028d00724e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.539858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.539829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdcf\" (UniqueName: \"kubernetes.io/projected/f84d1fa4-d0a3-4794-8e67-8f028d00724e-kube-api-access-8hdcf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl\" (UID: \"f84d1fa4-d0a3-4794-8e67-8f028d00724e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.754867 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.754777 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:19.915562 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:19.915510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl"] Jul 29 19:15:19.917503 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:15:19.917478 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf84d1fa4_d0a3_4794_8e67_8f028d00724e.slice/crio-c15fce55c3ea42d6033152dcf36de0755ea4815108aabc577ef5a3277655634c WatchSource:0}: Error finding container c15fce55c3ea42d6033152dcf36de0755ea4815108aabc577ef5a3277655634c: Status 404 returned error can't find the container with id c15fce55c3ea42d6033152dcf36de0755ea4815108aabc577ef5a3277655634c Jul 29 19:15:20.548593 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:20.548478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" event={"ID":"f84d1fa4-d0a3-4794-8e67-8f028d00724e","Type":"ContainerStarted","Data":"f86fc03b9ba35cddb57ae68fb28a0604c54ccb5f8cb279ed2953152f8fd7de21"} Jul 29 19:15:20.548593 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:20.548539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" event={"ID":"f84d1fa4-d0a3-4794-8e67-8f028d00724e","Type":"ContainerStarted","Data":"c15fce55c3ea42d6033152dcf36de0755ea4815108aabc577ef5a3277655634c"} Jul 29 19:15:20.567935 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:20.567880 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" podStartSLOduration=1.255980052 podStartE2EDuration="1.567865701s" totalImagesPullingTime="311.885649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:20.566181674 +0000 UTC m=+901.583202782" watchObservedRunningTime="2026-07-29 19:15:20.567865701 +0000 UTC m=+901.584886796" Jul 29 19:15:29.755851 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:29.755813 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:29.756399 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:29.755864 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:29.768653 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:29.768625 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:15:30.617043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:15:30.617015 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl" Jul 29 19:16:29.161024 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.160986 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-565f6d5884-w59bh"] Jul 29 19:16:29.164897 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.164866 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-565f6d5884-w59bh" Jul 29 19:16:29.168310 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.168281 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qllfn\"" Jul 29 19:16:29.168846 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.168804 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 19:16:29.179546 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.176261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-565f6d5884-w59bh"] Jul 29 19:16:29.296061 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.296012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wbg\" (UniqueName: \"kubernetes.io/projected/e36391c1-e0a0-441a-9392-9b73db31f7ab-kube-api-access-52wbg\") pod \"authorino-565f6d5884-w59bh\" (UID: \"e36391c1-e0a0-441a-9392-9b73db31f7ab\") " pod="kuadrant-system/authorino-565f6d5884-w59bh" Jul 29 19:16:29.296275 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.296099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e36391c1-e0a0-441a-9392-9b73db31f7ab-tls-cert\") pod \"authorino-565f6d5884-w59bh\" (UID: \"e36391c1-e0a0-441a-9392-9b73db31f7ab\") " pod="kuadrant-system/authorino-565f6d5884-w59bh" Jul 29 19:16:29.396990 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.396924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e36391c1-e0a0-441a-9392-9b73db31f7ab-tls-cert\") pod \"authorino-565f6d5884-w59bh\" (UID: \"e36391c1-e0a0-441a-9392-9b73db31f7ab\") " pod="kuadrant-system/authorino-565f6d5884-w59bh" Jul 29 19:16:29.397191 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.397077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wbg\" (UniqueName: \"kubernetes.io/projected/e36391c1-e0a0-441a-9392-9b73db31f7ab-kube-api-access-52wbg\") pod \"authorino-565f6d5884-w59bh\" (UID: \"e36391c1-e0a0-441a-9392-9b73db31f7ab\") " pod="kuadrant-system/authorino-565f6d5884-w59bh" Jul 29 19:16:29.399742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.399708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e36391c1-e0a0-441a-9392-9b73db31f7ab-tls-cert\") pod \"authorino-565f6d5884-w59bh\" (UID: \"e36391c1-e0a0-441a-9392-9b73db31f7ab\") " pod="kuadrant-system/authorino-565f6d5884-w59bh" Jul 29 19:16:29.405140 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.405107 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wbg\" (UniqueName: \"kubernetes.io/projected/e36391c1-e0a0-441a-9392-9b73db31f7ab-kube-api-access-52wbg\") pod \"authorino-565f6d5884-w59bh\" (UID: \"e36391c1-e0a0-441a-9392-9b73db31f7ab\") " pod="kuadrant-system/authorino-565f6d5884-w59bh" Jul 29 19:16:29.490662 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.490565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-565f6d5884-w59bh" Jul 29 19:16:29.642859 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.642832 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-565f6d5884-w59bh"] Jul 29 19:16:29.645432 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:16:29.645403 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode36391c1_e0a0_441a_9392_9b73db31f7ab.slice/crio-c328e881c509078e47ab513a87f32990b12959fc9bedf8928e9d8bccffcd588d WatchSource:0}: Error finding container c328e881c509078e47ab513a87f32990b12959fc9bedf8928e9d8bccffcd588d: Status 404 returned error can't find the container with id c328e881c509078e47ab513a87f32990b12959fc9bedf8928e9d8bccffcd588d Jul 29 19:16:29.646731 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.646711 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:16:29.948165 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:29.948127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-565f6d5884-w59bh" event={"ID":"e36391c1-e0a0-441a-9392-9b73db31f7ab","Type":"ContainerStarted","Data":"c328e881c509078e47ab513a87f32990b12959fc9bedf8928e9d8bccffcd588d"} Jul 29 19:16:30.957026 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:30.956979 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-565f6d5884-w59bh" event={"ID":"e36391c1-e0a0-441a-9392-9b73db31f7ab","Type":"ContainerStarted","Data":"ace0adf5f22fe0947e5787b5e5ebc83f431b51a0e17ceec07c90f15e8a359fcf"} Jul 29 19:16:30.976825 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:16:30.976771 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-565f6d5884-w59bh" podStartSLOduration=1.539044914 podStartE2EDuration="1.976750405s" totalImagesPullingTime="437.705491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:16:30.976178208 +0000 UTC m=+971.993199304" watchObservedRunningTime="2026-07-29 19:16:30.976750405 +0000 UTC m=+971.993771500" Jul 29 19:17:16.750808 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:16.750769 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-vxv8r"] Jul 29 19:17:16.751346 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:16.751056 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" podUID="8d2a6b7f-e26a-4728-af0d-2163abe5c68f" containerName="manager" containerID="cri-o://008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2" gracePeriod=30 Jul 29 19:17:17.027251 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.027216 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:17:17.163000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.162963 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-webhook-cert\") pod \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\" (UID: \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\") " Jul 29 19:17:17.163172 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.163084 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb7lf\" (UniqueName: \"kubernetes.io/projected/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-kube-api-access-mb7lf\") pod \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\" (UID: \"8d2a6b7f-e26a-4728-af0d-2163abe5c68f\") " Jul 29 19:17:17.165149 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.165115 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8d2a6b7f-e26a-4728-af0d-2163abe5c68f" (UID: "8d2a6b7f-e26a-4728-af0d-2163abe5c68f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:17:17.165149 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.165118 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-kube-api-access-mb7lf" (OuterVolumeSpecName: "kube-api-access-mb7lf") pod "8d2a6b7f-e26a-4728-af0d-2163abe5c68f" (UID: "8d2a6b7f-e26a-4728-af0d-2163abe5c68f"). InnerVolumeSpecName "kube-api-access-mb7lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:17:17.242602 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.242563 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d2a6b7f-e26a-4728-af0d-2163abe5c68f" containerID="008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2" exitCode=0 Jul 29 19:17:17.242602 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.242604 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" Jul 29 19:17:17.242839 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.242652 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" event={"ID":"8d2a6b7f-e26a-4728-af0d-2163abe5c68f","Type":"ContainerDied","Data":"008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2"} Jul 29 19:17:17.242839 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.242711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-vxv8r" event={"ID":"8d2a6b7f-e26a-4728-af0d-2163abe5c68f","Type":"ContainerDied","Data":"2bbd69465e2e723e638075cfa72f8663b20f285a705843f78ca699934f08821a"} Jul 29 19:17:17.242839 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.242734 2575 scope.go:117] "RemoveContainer" containerID="008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2" Jul 29 19:17:17.254299 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.254281 2575 scope.go:117] "RemoveContainer" containerID="008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2" Jul 29 19:17:17.254616 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:17:17.254594 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2\": container with ID starting with 008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2 not found: ID does not exist" containerID="008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2" Jul 29 19:17:17.254715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.254624 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2"} err="failed to get container status \"008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2\": rpc error: code = NotFound desc = could not find container \"008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2\": container with ID starting with 008c133a3035714ccd1bb7edd99acb42455e5b082557ce7fa354aec7d980ece2 not found: ID does not exist" Jul 29 19:17:17.264565 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.264493 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-webhook-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:17:17.264565 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.264561 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7lf\" (UniqueName: \"kubernetes.io/projected/8d2a6b7f-e26a-4728-af0d-2163abe5c68f-kube-api-access-mb7lf\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:17:17.274263 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.274227 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-vxv8r"] Jul 29 19:17:17.285087 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.285047 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-vxv8r"] Jul 29 19:17:17.550467 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:17:17.550379 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d2a6b7f-e26a-4728-af0d-2163abe5c68f" path="/var/lib/kubelet/pods/8d2a6b7f-e26a-4728-af0d-2163abe5c68f/volumes" Jul 29 19:18:18.133573 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.133478 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-cxjmf"] Jul 29 19:18:18.134020 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.133900 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2a6b7f-e26a-4728-af0d-2163abe5c68f" containerName="manager" Jul 29 19:18:18.134020 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.133911 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2a6b7f-e26a-4728-af0d-2163abe5c68f" containerName="manager" Jul 29 19:18:18.134020 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.133995 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2a6b7f-e26a-4728-af0d-2163abe5c68f" containerName="manager" Jul 29 19:18:18.137168 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.137145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:18.140615 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.140591 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zgvq9\"" Jul 29 19:18:18.141354 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.141337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 19:18:18.141676 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.141658 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 19:18:18.153841 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.153810 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-cxjmf"] Jul 29 19:18:18.242222 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.242171 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-webhook-cert\") pod \"maas-controller-7c6f6796c6-cxjmf\" (UID: \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\") " pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:18.242411 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.242311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjcvr\" (UniqueName: \"kubernetes.io/projected/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-kube-api-access-kjcvr\") pod \"maas-controller-7c6f6796c6-cxjmf\" (UID: \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\") " pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:18.343465 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.343425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjcvr\" (UniqueName: \"kubernetes.io/projected/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-kube-api-access-kjcvr\") pod \"maas-controller-7c6f6796c6-cxjmf\" (UID: \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\") " pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:18.343664 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.343543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-webhook-cert\") pod \"maas-controller-7c6f6796c6-cxjmf\" (UID: \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\") " pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:18.345934 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.345910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-webhook-cert\") pod \"maas-controller-7c6f6796c6-cxjmf\" (UID: \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\") " pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:18.357860 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.357823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjcvr\" (UniqueName: \"kubernetes.io/projected/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-kube-api-access-kjcvr\") pod \"maas-controller-7c6f6796c6-cxjmf\" (UID: \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\") " pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:18.448359 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.448264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:18.600965 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.600932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-cxjmf"] Jul 29 19:18:18.602624 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:18:18.602587 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1ba4c1_3a90_4832_861b_e31a8ca20c57.slice/crio-4af0904c6469432940e47e61c6dac257d73e04fc442ad9fa7969bf243eb6d368 WatchSource:0}: Error finding container 4af0904c6469432940e47e61c6dac257d73e04fc442ad9fa7969bf243eb6d368: Status 404 returned error can't find the container with id 4af0904c6469432940e47e61c6dac257d73e04fc442ad9fa7969bf243eb6d368 Jul 29 19:18:18.609771 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:18.609724 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" event={"ID":"bd1ba4c1-3a90-4832-861b-e31a8ca20c57","Type":"ContainerStarted","Data":"4af0904c6469432940e47e61c6dac257d73e04fc442ad9fa7969bf243eb6d368"} Jul 29 19:18:19.617415 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:19.617298 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" event={"ID":"bd1ba4c1-3a90-4832-861b-e31a8ca20c57","Type":"ContainerStarted","Data":"6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7"} Jul 29 19:18:19.617415 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:19.617378 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:19.636270 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:19.636197 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" podStartSLOduration=1.109708246 podStartE2EDuration="1.636178394s" totalImagesPullingTime="526.470148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:19.634582389 +0000 UTC m=+1080.651603485" watchObservedRunningTime="2026-07-29 19:18:19.636178394 +0000 UTC m=+1080.653199491" Jul 29 19:18:30.625289 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:30.625256 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:35.258556 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.258500 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-cxjmf"] Jul 29 19:18:35.259088 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.258721 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" podUID="bd1ba4c1-3a90-4832-861b-e31a8ca20c57" containerName="manager" containerID="cri-o://6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7" gracePeriod=30 Jul 29 19:18:35.516351 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.516280 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:35.617675 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.617632 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjcvr\" (UniqueName: \"kubernetes.io/projected/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-kube-api-access-kjcvr\") pod \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\" (UID: \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\") " Jul 29 19:18:35.617843 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.617708 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-webhook-cert\") pod \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\" (UID: \"bd1ba4c1-3a90-4832-861b-e31a8ca20c57\") " Jul 29 19:18:35.619817 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.619785 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bd1ba4c1-3a90-4832-861b-e31a8ca20c57" (UID: "bd1ba4c1-3a90-4832-861b-e31a8ca20c57"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:18:35.619942 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.619854 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-kube-api-access-kjcvr" (OuterVolumeSpecName: "kube-api-access-kjcvr") pod "bd1ba4c1-3a90-4832-861b-e31a8ca20c57" (UID: "bd1ba4c1-3a90-4832-861b-e31a8ca20c57"). InnerVolumeSpecName "kube-api-access-kjcvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:18:35.715450 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.715414 2575 generic.go:356] "Generic (PLEG): container finished" podID="bd1ba4c1-3a90-4832-861b-e31a8ca20c57" containerID="6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7" exitCode=0 Jul 29 19:18:35.715667 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.715466 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" Jul 29 19:18:35.715667 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.715496 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" event={"ID":"bd1ba4c1-3a90-4832-861b-e31a8ca20c57","Type":"ContainerDied","Data":"6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7"} Jul 29 19:18:35.715667 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.715578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-cxjmf" event={"ID":"bd1ba4c1-3a90-4832-861b-e31a8ca20c57","Type":"ContainerDied","Data":"4af0904c6469432940e47e61c6dac257d73e04fc442ad9fa7969bf243eb6d368"} Jul 29 19:18:35.715667 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.715601 2575 scope.go:117] "RemoveContainer" containerID="6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7" Jul 29 19:18:35.718523 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.718485 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjcvr\" (UniqueName: \"kubernetes.io/projected/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-kube-api-access-kjcvr\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:18:35.718672 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.718506 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd1ba4c1-3a90-4832-861b-e31a8ca20c57-webhook-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:18:35.725872 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.725851 2575 scope.go:117] "RemoveContainer" containerID="6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7" Jul 29 19:18:35.726204 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:18:35.726184 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7\": container with ID starting with 6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7 not found: ID does not exist" containerID="6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7" Jul 29 19:18:35.726263 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.726214 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7"} err="failed to get container status \"6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7\": rpc error: code = NotFound desc = could not find container \"6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7\": container with ID starting with 6fcd3376f018864d29cf2b7c5f307fd1277f8e031d061844f197f322606ffbf7 not found: ID does not exist" Jul 29 19:18:35.738762 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.738722 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-cxjmf"] Jul 29 19:18:35.742614 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:35.742569 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-cxjmf"] Jul 29 19:18:36.559999 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.559960 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-74wqj"] Jul 29 19:18:36.560413 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.560379 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd1ba4c1-3a90-4832-861b-e31a8ca20c57" containerName="manager" Jul 29 19:18:36.560413 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.560390 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd1ba4c1-3a90-4832-861b-e31a8ca20c57" containerName="manager" Jul 29 19:18:36.560491 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.560462 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd1ba4c1-3a90-4832-861b-e31a8ca20c57" containerName="manager" Jul 29 19:18:36.565049 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.565024 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:36.569130 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.569101 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 19:18:36.569318 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.569281 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 19:18:36.569641 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.569108 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zgvq9\"" Jul 29 19:18:36.573331 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.573293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-74wqj"] Jul 29 19:18:36.729029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.728982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b03c7525-db2b-41d8-b372-7cee498fb6ba-webhook-cert\") pod \"maas-controller-7c6f6796c6-74wqj\" (UID: \"b03c7525-db2b-41d8-b372-7cee498fb6ba\") " pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:36.729029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.729029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjk89\" (UniqueName: \"kubernetes.io/projected/b03c7525-db2b-41d8-b372-7cee498fb6ba-kube-api-access-tjk89\") pod \"maas-controller-7c6f6796c6-74wqj\" (UID: \"b03c7525-db2b-41d8-b372-7cee498fb6ba\") " pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:36.830475 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.830371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b03c7525-db2b-41d8-b372-7cee498fb6ba-webhook-cert\") pod \"maas-controller-7c6f6796c6-74wqj\" (UID: \"b03c7525-db2b-41d8-b372-7cee498fb6ba\") " pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:36.830475 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.830410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjk89\" (UniqueName: \"kubernetes.io/projected/b03c7525-db2b-41d8-b372-7cee498fb6ba-kube-api-access-tjk89\") pod \"maas-controller-7c6f6796c6-74wqj\" (UID: \"b03c7525-db2b-41d8-b372-7cee498fb6ba\") " pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:36.832921 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.832892 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b03c7525-db2b-41d8-b372-7cee498fb6ba-webhook-cert\") pod \"maas-controller-7c6f6796c6-74wqj\" (UID: \"b03c7525-db2b-41d8-b372-7cee498fb6ba\") " pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:36.839153 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.839121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjk89\" (UniqueName: \"kubernetes.io/projected/b03c7525-db2b-41d8-b372-7cee498fb6ba-kube-api-access-tjk89\") pod \"maas-controller-7c6f6796c6-74wqj\" (UID: \"b03c7525-db2b-41d8-b372-7cee498fb6ba\") " pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:36.881177 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:36.881126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:37.028813 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:37.028781 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c6f6796c6-74wqj"] Jul 29 19:18:37.030872 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:18:37.030845 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03c7525_db2b_41d8_b372_7cee498fb6ba.slice/crio-6c7d703e109d0bc6e5c0dc3a7e2dca165a94b341f48f4b001bbfe945ae848068 WatchSource:0}: Error finding container 6c7d703e109d0bc6e5c0dc3a7e2dca165a94b341f48f4b001bbfe945ae848068: Status 404 returned error can't find the container with id 6c7d703e109d0bc6e5c0dc3a7e2dca165a94b341f48f4b001bbfe945ae848068 Jul 29 19:18:37.551552 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:37.551496 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd1ba4c1-3a90-4832-861b-e31a8ca20c57" path="/var/lib/kubelet/pods/bd1ba4c1-3a90-4832-861b-e31a8ca20c57/volumes" Jul 29 19:18:37.730245 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:37.730210 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-74wqj" event={"ID":"b03c7525-db2b-41d8-b372-7cee498fb6ba","Type":"ContainerStarted","Data":"d8b1ae7ea1c3ad88798db991ff6d890144fe0f5a983f0618620c946a90de0d6e"} Jul 29 19:18:37.730680 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:37.730263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c6f6796c6-74wqj" event={"ID":"b03c7525-db2b-41d8-b372-7cee498fb6ba","Type":"ContainerStarted","Data":"6c7d703e109d0bc6e5c0dc3a7e2dca165a94b341f48f4b001bbfe945ae848068"} Jul 29 19:18:37.730680 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:37.730328 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:18:37.755082 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:37.754960 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c6f6796c6-74wqj" podStartSLOduration=1.286316043 podStartE2EDuration="1.754937788s" totalImagesPullingTime="468.621745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:37.751968279 +0000 UTC m=+1098.768989376" watchObservedRunningTime="2026-07-29 19:18:37.754937788 +0000 UTC m=+1098.771958885" Jul 29 19:18:48.738257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:18:48.738226 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c6f6796c6-74wqj" Jul 29 19:23:49.973186 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:49.973146 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2"] Jul 29 19:23:49.973795 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:49.973380 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" podUID="2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a" containerName="manager" containerID="cri-o://8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784" gracePeriod=10 Jul 29 19:23:54.129840 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.129817 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:23:54.207728 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.207647 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-extensions-socket-volume\") pod \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\" (UID: \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\") " Jul 29 19:23:54.207875 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.207784 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zzfh\" (UniqueName: \"kubernetes.io/projected/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-kube-api-access-6zzfh\") pod \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\" (UID: \"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a\") " Jul 29 19:23:54.207997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.207975 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a" (UID: "2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:54.209673 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.209648 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-kube-api-access-6zzfh" (OuterVolumeSpecName: "kube-api-access-6zzfh") pod "2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a" (UID: "2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a"). InnerVolumeSpecName "kube-api-access-6zzfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:23:54.308858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.308814 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zzfh\" (UniqueName: \"kubernetes.io/projected/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-kube-api-access-6zzfh\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:23:54.308858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.308854 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a-extensions-socket-volume\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:23:54.616240 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.616158 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a" containerID="8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784" exitCode=0 Jul 29 19:23:54.616240 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.616200 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" Jul 29 19:23:54.616418 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.616237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" event={"ID":"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a","Type":"ContainerDied","Data":"8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784"} Jul 29 19:23:54.616418 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.616288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2" event={"ID":"2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a","Type":"ContainerDied","Data":"1d5dd15e7d07eb95465fe6b05b54d337ca05928b1698b9cac4c905102caabdc3"} Jul 29 19:23:54.616418 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.616312 2575 scope.go:117] "RemoveContainer" containerID="8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784" Jul 29 19:23:54.626613 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.626596 2575 scope.go:117] "RemoveContainer" containerID="8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784" Jul 29 19:23:54.626858 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:23:54.626840 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784\": container with ID starting with 8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784 not found: ID does not exist" containerID="8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784" Jul 29 19:23:54.626903 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.626867 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784"} err="failed to get container status \"8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784\": rpc error: code = NotFound desc = could not find container \"8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784\": container with ID starting with 8628e990850f97af8bd9e8ac2accf9a223224771c5dd7b4315a9199e61d0d784 not found: ID does not exist" Jul 29 19:23:54.638434 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.638413 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2"] Jul 29 19:23:54.642482 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:54.642460 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wd4r2"] Jul 29 19:23:55.549480 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:23:55.549446 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a" path="/var/lib/kubelet/pods/2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a/volumes" Jul 29 19:24:51.113606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.113567 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82"] Jul 29 19:24:51.114083 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.113997 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a" containerName="manager" Jul 29 19:24:51.114083 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.114009 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a" containerName="manager" Jul 29 19:24:51.114159 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.114118 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e99e69d-cf24-4ff8-ba43-caeb3bb5f70a" containerName="manager" Jul 29 19:24:51.117077 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.117061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.119802 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.119776 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-879x4\"" Jul 29 19:24:51.142575 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.142549 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82"] Jul 29 19:24:51.220287 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.220247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhml6\" (UniqueName: \"kubernetes.io/projected/ea2c139c-5793-4283-8dee-60b33a8136eb-kube-api-access-bhml6\") pod \"kuadrant-operator-controller-manager-86878c769f-2vj82\" (UID: \"ea2c139c-5793-4283-8dee-60b33a8136eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.220446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.220316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea2c139c-5793-4283-8dee-60b33a8136eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2vj82\" (UID: \"ea2c139c-5793-4283-8dee-60b33a8136eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.321785 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.321745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea2c139c-5793-4283-8dee-60b33a8136eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2vj82\" (UID: \"ea2c139c-5793-4283-8dee-60b33a8136eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.321945 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.321832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhml6\" (UniqueName: \"kubernetes.io/projected/ea2c139c-5793-4283-8dee-60b33a8136eb-kube-api-access-bhml6\") pod \"kuadrant-operator-controller-manager-86878c769f-2vj82\" (UID: \"ea2c139c-5793-4283-8dee-60b33a8136eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.322201 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.322179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea2c139c-5793-4283-8dee-60b33a8136eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2vj82\" (UID: \"ea2c139c-5793-4283-8dee-60b33a8136eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.332295 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.332271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhml6\" (UniqueName: \"kubernetes.io/projected/ea2c139c-5793-4283-8dee-60b33a8136eb-kube-api-access-bhml6\") pod \"kuadrant-operator-controller-manager-86878c769f-2vj82\" (UID: \"ea2c139c-5793-4283-8dee-60b33a8136eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.427606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.427575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.610784 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:24:51.610753 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2c139c_5793_4283_8dee_60b33a8136eb.slice/crio-8aae4df812981e01d51741d5faa813beed06152e5c6179f7e263b1b48ea62d88 WatchSource:0}: Error finding container 8aae4df812981e01d51741d5faa813beed06152e5c6179f7e263b1b48ea62d88: Status 404 returned error can't find the container with id 8aae4df812981e01d51741d5faa813beed06152e5c6179f7e263b1b48ea62d88 Jul 29 19:24:51.611209 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.611186 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82"] Jul 29 19:24:51.928509 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.928474 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" event={"ID":"ea2c139c-5793-4283-8dee-60b33a8136eb","Type":"ContainerStarted","Data":"239d359e0070330a010cdf1252f47ec399224c29938226e766b76b845a742ac7"} Jul 29 19:24:51.928730 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.928549 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:24:51.928730 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.928569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" event={"ID":"ea2c139c-5793-4283-8dee-60b33a8136eb","Type":"ContainerStarted","Data":"8aae4df812981e01d51741d5faa813beed06152e5c6179f7e263b1b48ea62d88"} Jul 29 19:24:51.960772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:24:51.960722 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" podStartSLOduration=0.960704079 podStartE2EDuration="960.704079ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:24:51.957431506 +0000 UTC m=+1472.974452603" watchObservedRunningTime="2026-07-29 19:24:51.960704079 +0000 UTC m=+1472.977725173" Jul 29 19:25:02.936865 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:25:02.936834 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2vj82" Jul 29 19:31:50.291571 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.291487 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn"] Jul 29 19:31:50.294837 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.294821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.297608 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.297588 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a448180e-openshift-default-dockercfg-x8rvz\"" Jul 29 19:31:50.335245 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.335220 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn"] Jul 29 19:31:50.414156 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-socket\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.414340 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-data\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.414340 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-certs\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.414340 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a147557f-9d99-41ea-affb-e33913311109-istiod-ca-cert\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.414340 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44977\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-kube-api-access-44977\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.414340 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-istio-token\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.414557 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-envoy\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.414557 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-credential-socket\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.414557 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.414433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a147557f-9d99-41ea-affb-e33913311109-istio-podinfo\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.515738 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-certs\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.515738 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a147557f-9d99-41ea-affb-e33913311109-istiod-ca-cert\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44977\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-kube-api-access-44977\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-istio-token\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-envoy\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-credential-socket\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a147557f-9d99-41ea-affb-e33913311109-istio-podinfo\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-socket\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.515984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-data\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516352 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.516151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-certs\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516352 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.516333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-credential-socket\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516452 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.516386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-data\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516508 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.516456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-socket\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.516604 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.516563 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a147557f-9d99-41ea-affb-e33913311109-istiod-ca-cert\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.518279 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.518252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-envoy\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.518391 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.518365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a147557f-9d99-41ea-affb-e33913311109-istio-podinfo\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.528053 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.528030 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44977\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-kube-api-access-44977\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.528179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.528053 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-istio-token\") pod \"e2e-ait-a448180e-openshift-default-5664986798-cphnn\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.604994 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.604899 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:50.753237 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.753211 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn"] Jul 29 19:31:50.755137 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:31:50.755111 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda147557f_9d99_41ea_affb_e33913311109.slice/crio-f2ad29d27f69599cbf6020ed2d9538558993c32da3ad163fcdef5993522d9c59 WatchSource:0}: Error finding container f2ad29d27f69599cbf6020ed2d9538558993c32da3ad163fcdef5993522d9c59: Status 404 returned error can't find the container with id f2ad29d27f69599cbf6020ed2d9538558993c32da3ad163fcdef5993522d9c59 Jul 29 19:31:50.757331 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.757289 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:31:50.757395 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.757372 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:31:50.757429 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:50.757406 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:31:51.337725 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:51.337684 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" event={"ID":"a147557f-9d99-41ea-affb-e33913311109","Type":"ContainerStarted","Data":"8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79"} Jul 29 19:31:51.338138 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:51.337749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" event={"ID":"a147557f-9d99-41ea-affb-e33913311109","Type":"ContainerStarted","Data":"f2ad29d27f69599cbf6020ed2d9538558993c32da3ad163fcdef5993522d9c59"} Jul 29 19:31:51.365446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:51.365384 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" podStartSLOduration=1.365367789 podStartE2EDuration="1.365367789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:51.364218077 +0000 UTC m=+1892.381239167" watchObservedRunningTime="2026-07-29 19:31:51.365367789 +0000 UTC m=+1892.382388882" Jul 29 19:31:51.605286 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:51.605186 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:51.610235 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:51.610209 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:52.343752 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:52.343720 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:31:52.344639 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:31:52.344620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:32:06.423751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:06.423719 2575 generic.go:356] "Generic (PLEG): container finished" podID="a147557f-9d99-41ea-affb-e33913311109" containerID="8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79" exitCode=0 Jul 29 19:32:06.424149 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:06.423772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" event={"ID":"a147557f-9d99-41ea-affb-e33913311109","Type":"ContainerDied","Data":"8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79"} Jul 29 19:32:06.424206 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:06.424151 2575 scope.go:117] "RemoveContainer" containerID="8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79" Jul 29 19:32:06.425186 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:06.425155 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:06.425256 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:06.425227 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:06.425324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:06.425310 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:07.344498 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:07.344422 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:32:07.431981 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:07.431941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" event={"ID":"a147557f-9d99-41ea-affb-e33913311109","Type":"ContainerStarted","Data":"06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451"} Jul 29 19:32:07.605446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:07.605361 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:32:07.606608 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:07.606579 2575 patch_prober.go:28] interesting pod/e2e-ait-a448180e-openshift-default-5664986798-cphnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:32:07.606737 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:07.606637 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 29 19:32:08.606008 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:08.605970 2575 patch_prober.go:28] interesting pod/e2e-ait-a448180e-openshift-default-5664986798-cphnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:32:08.606393 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:08.606027 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 29 19:32:09.606286 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:09.606241 2575 patch_prober.go:28] interesting pod/e2e-ait-a448180e-openshift-default-5664986798-cphnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:32:09.606803 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:09.606318 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 29 19:32:10.605912 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:10.605875 2575 patch_prober.go:28] interesting pod/e2e-ait-a448180e-openshift-default-5664986798-cphnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:32:10.606107 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:10.605939 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 29 19:32:11.609726 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:11.609694 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:32:11.610198 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:11.609983 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:32:11.610775 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:11.610743 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:32:23.794239 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:23.794202 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn"] Jul 29 19:32:23.794775 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:23.794478 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" containerID="cri-o://06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451" gracePeriod=30 Jul 29 19:32:24.819975 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.819944 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9"] Jul 29 19:32:24.822718 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.822699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.825945 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.825921 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e80635b4-gw-openshift-default-dockercfg-6gwnl\"" Jul 29 19:32:24.836803 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.836777 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9"] Jul 29 19:32:24.946302 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-podinfo\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.946487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-data\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.946487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946351 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-token\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.946487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istiod-ca-cert\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.946666 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-envoy\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.946711 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-credential-socket\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.946758 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946713 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-certs\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.946816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946801 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sgm\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-kube-api-access-k8sgm\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:24.946863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:24.946846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-socket\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.047916 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.047877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sgm\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-kube-api-access-k8sgm\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048107 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.047925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-socket\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048107 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-podinfo\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048231 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-data\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048231 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-token\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048231 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istiod-ca-cert\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048231 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-envoy\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-socket\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-credential-socket\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-certs\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048616 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-data\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048687 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-certs\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.048734 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.048688 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-credential-socket\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.049084 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.049055 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istiod-ca-cert\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.050492 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.050466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-envoy\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.050741 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.050723 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-podinfo\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.057399 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.057378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-token\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.057715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.057697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sgm\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-kube-api-access-k8sgm\") pod \"e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.132680 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.132657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:25.284685 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.284659 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9"] Jul 29 19:32:25.289834 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:32:25.289772 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ae4ccf_e770_44dc_8b33_8fc061ff1403.slice/crio-f661b602ae82e0e83a9b984593649c195b0782125a4f7022bfbc6ac7fc202246 WatchSource:0}: Error finding container f661b602ae82e0e83a9b984593649c195b0782125a4f7022bfbc6ac7fc202246: Status 404 returned error can't find the container with id f661b602ae82e0e83a9b984593649c195b0782125a4f7022bfbc6ac7fc202246 Jul 29 19:32:25.291595 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.291554 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:25.291707 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.291645 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:25.291707 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.291691 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:25.553987 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.553899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" event={"ID":"91ae4ccf-e770-44dc-8b33-8fc061ff1403","Type":"ContainerStarted","Data":"041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53"} Jul 29 19:32:25.553987 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.553954 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" event={"ID":"91ae4ccf-e770-44dc-8b33-8fc061ff1403","Type":"ContainerStarted","Data":"f661b602ae82e0e83a9b984593649c195b0782125a4f7022bfbc6ac7fc202246"} Jul 29 19:32:25.574972 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:25.574923 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" podStartSLOduration=1.574906991 podStartE2EDuration="1.574906991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:32:25.573492438 +0000 UTC m=+1926.590513531" watchObservedRunningTime="2026-07-29 19:32:25.574906991 +0000 UTC m=+1926.591928086" Jul 29 19:32:26.133799 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:26.133765 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:26.138843 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:26.138817 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:26.560048 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:26.559965 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:26.560923 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:26.560904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:26.612895 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:26.612856 2575 patch_prober.go:28] interesting pod/e2e-ait-a448180e-openshift-default-5664986798-cphnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 29 19:32:26.613075 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:26.612929 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:32:29.048207 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.048182 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:32:29.088850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.088768 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-istio-token\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.088850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.088834 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-socket\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.089073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.088856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-certs\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.089073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.088908 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-envoy\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.089073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.088977 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44977\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-kube-api-access-44977\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.089073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.089007 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-credential-socket\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.089073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.089061 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-data\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.089350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.089091 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a147557f-9d99-41ea-affb-e33913311109-istio-podinfo\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.089350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.089119 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a147557f-9d99-41ea-affb-e33913311109-istiod-ca-cert\") pod \"a147557f-9d99-41ea-affb-e33913311109\" (UID: \"a147557f-9d99-41ea-affb-e33913311109\") " Jul 29 19:32:29.089350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.089134 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:32:29.089350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.089206 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:32:29.089350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.089345 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:32:29.089945 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.089907 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a147557f-9d99-41ea-affb-e33913311109-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:32:29.090056 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.090023 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-data" (OuterVolumeSpecName: "istio-data") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:32:29.090190 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.090169 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.090389 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.090368 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.090488 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.090392 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.090488 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.090407 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.090488 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.090420 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a147557f-9d99-41ea-affb-e33913311109-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.091377 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.091357 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-istio-token" (OuterVolumeSpecName: "istio-token") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:32:29.091651 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.091627 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:32:29.091761 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.091629 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-kube-api-access-44977" (OuterVolumeSpecName: "kube-api-access-44977") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "kube-api-access-44977". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:32:29.091833 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.091810 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a147557f-9d99-41ea-affb-e33913311109-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a147557f-9d99-41ea-affb-e33913311109" (UID: "a147557f-9d99-41ea-affb-e33913311109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:32:29.191715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.191683 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a147557f-9d99-41ea-affb-e33913311109-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.191715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.191710 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44977\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-kube-api-access-44977\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.191715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.191721 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a147557f-9d99-41ea-affb-e33913311109-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.191715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.191729 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a147557f-9d99-41ea-affb-e33913311109-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:32:29.580441 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.580403 2575 generic.go:356] "Generic (PLEG): container finished" podID="a147557f-9d99-41ea-affb-e33913311109" containerID="06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451" exitCode=0 Jul 29 19:32:29.580622 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.580451 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" Jul 29 19:32:29.580622 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.580483 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" event={"ID":"a147557f-9d99-41ea-affb-e33913311109","Type":"ContainerDied","Data":"06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451"} Jul 29 19:32:29.580622 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.580553 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn" event={"ID":"a147557f-9d99-41ea-affb-e33913311109","Type":"ContainerDied","Data":"f2ad29d27f69599cbf6020ed2d9538558993c32da3ad163fcdef5993522d9c59"} Jul 29 19:32:29.580622 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.580578 2575 scope.go:117] "RemoveContainer" containerID="06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451" Jul 29 19:32:29.589769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.589740 2575 scope.go:117] "RemoveContainer" containerID="8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79" Jul 29 19:32:29.597982 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.597964 2575 scope.go:117] "RemoveContainer" containerID="06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451" Jul 29 19:32:29.598214 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:32:29.598195 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451\": container with ID starting with 06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451 not found: ID does not exist" containerID="06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451" Jul 29 19:32:29.598255 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.598223 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451"} err="failed to get container status \"06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451\": rpc error: code = NotFound desc = could not find container \"06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451\": container with ID starting with 06a97a694e8190946b8a6f173e9307e398c1523071e9f0787236f22177bd8451 not found: ID does not exist" Jul 29 19:32:29.598255 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.598245 2575 scope.go:117] "RemoveContainer" containerID="8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79" Jul 29 19:32:29.598469 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:32:29.598421 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79\": container with ID starting with 8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79 not found: ID does not exist" containerID="8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79" Jul 29 19:32:29.598469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.598441 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79"} err="failed to get container status \"8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79\": rpc error: code = NotFound desc = could not find container \"8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79\": container with ID starting with 8b0fd39638d7550931a25a805a0c9d5734f677ac47c1ed5705c01a401fdb7e79 not found: ID does not exist" Jul 29 19:32:29.601953 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.601931 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn"] Jul 29 19:32:29.605579 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:29.605559 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a448180e-openshift-default-5664986798-cphnn"] Jul 29 19:32:31.549652 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:31.549619 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a147557f-9d99-41ea-affb-e33913311109" path="/var/lib/kubelet/pods/a147557f-9d99-41ea-affb-e33913311109/volumes" Jul 29 19:32:41.631771 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:41.631742 2575 patch_prober.go:28] interesting pod/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:52624->10.132.0.61:15021: read: connection reset by peer" start-of-body= Jul 29 19:32:41.632183 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:41.631823 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:52624->10.132.0.61:15021: read: connection reset by peer" Jul 29 19:32:42.663379 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:42.663345 2575 generic.go:356] "Generic (PLEG): container finished" podID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerID="041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53" exitCode=0 Jul 29 19:32:42.663379 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:42.663387 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" event={"ID":"91ae4ccf-e770-44dc-8b33-8fc061ff1403","Type":"ContainerDied","Data":"041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53"} Jul 29 19:32:42.663924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:42.663896 2575 scope.go:117] "RemoveContainer" containerID="041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53" Jul 29 19:32:42.664967 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:42.664933 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:42.665078 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:42.665003 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:42.665078 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:42.665036 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:43.670597 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:43.670561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" event={"ID":"91ae4ccf-e770-44dc-8b33-8fc061ff1403","Type":"ContainerStarted","Data":"5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b"} Jul 29 19:32:44.133623 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:44.133591 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:44.133797 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:44.133774 2575 patch_prober.go:28] interesting pod/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 29 19:32:44.133857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:44.133839 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 29 19:32:45.133759 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:45.133720 2575 patch_prober.go:28] interesting pod/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 29 19:32:45.134262 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:45.133785 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 29 19:32:46.133318 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:46.133275 2575 patch_prober.go:28] interesting pod/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 29 19:32:46.133544 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:46.133343 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 29 19:32:47.133223 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:47.133185 2575 patch_prober.go:28] interesting pod/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 29 19:32:47.133655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:47.133240 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 29 19:32:48.137627 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:48.137589 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:48.138024 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:48.137994 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:48.138633 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:48.138614 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:32:58.103785 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.103752 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9"] Jul 29 19:32:58.104384 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.104013 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" containerID="cri-o://5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b" gracePeriod=30 Jul 29 19:32:58.949458 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.949418 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6"] Jul 29 19:32:58.949919 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.949903 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" Jul 29 19:32:58.949964 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.949920 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" Jul 29 19:32:58.949964 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.949935 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" Jul 29 19:32:58.949964 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.949941 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" Jul 29 19:32:58.950060 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.950016 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" Jul 29 19:32:58.950060 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.950025 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a147557f-9d99-41ea-affb-e33913311109" containerName="istio-proxy" Jul 29 19:32:58.952227 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.952211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.955219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.955190 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6d42a5b6-openshift-default-dockercfg-k8vvb\"" Jul 29 19:32:58.968987 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.968952 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6"] Jul 29 19:32:58.986233 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-socket\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.986393 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-credential-socket\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.986393 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-certs\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.986487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b88b09-ec86-4812-986d-f98af3fd2a1e-istiod-ca-cert\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.986487 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-podinfo\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.986602 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-data\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.986602 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82n2\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-kube-api-access-w82n2\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.986683 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-envoy\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:58.986683 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:58.986658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-token\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088209 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-envoy\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-token\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-socket\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-credential-socket\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-certs\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b88b09-ec86-4812-986d-f98af3fd2a1e-istiod-ca-cert\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-podinfo\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-data\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82n2\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-kube-api-access-w82n2\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.088855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088808 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-certs\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.089009 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.088971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-data\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.089110 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.089002 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-credential-socket\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.089172 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.089111 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-socket\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.089467 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.089443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b88b09-ec86-4812-986d-f98af3fd2a1e-istiod-ca-cert\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.090698 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.090667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-envoy\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.090971 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.090952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-podinfo\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.097276 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.097249 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-token\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.097276 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.097261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82n2\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-kube-api-access-w82n2\") pod \"e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.263816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.263729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:32:59.405645 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.405618 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6"] Jul 29 19:32:59.406877 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:32:59.406850 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b88b09_ec86_4812_986d_f98af3fd2a1e.slice/crio-e7c2a827dab9175aea76666bdb31209db9f824149ff45538e30f8336302a1915 WatchSource:0}: Error finding container e7c2a827dab9175aea76666bdb31209db9f824149ff45538e30f8336302a1915: Status 404 returned error can't find the container with id e7c2a827dab9175aea76666bdb31209db9f824149ff45538e30f8336302a1915 Jul 29 19:32:59.409162 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.409132 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:59.409230 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.409196 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:59.409230 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.409225 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:32:59.764382 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.764345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" event={"ID":"75b88b09-ec86-4812-986d-f98af3fd2a1e","Type":"ContainerStarted","Data":"af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb"} Jul 29 19:32:59.764582 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.764404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" event={"ID":"75b88b09-ec86-4812-986d-f98af3fd2a1e","Type":"ContainerStarted","Data":"e7c2a827dab9175aea76666bdb31209db9f824149ff45538e30f8336302a1915"} Jul 29 19:32:59.785431 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:32:59.785382 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" podStartSLOduration=1.785367258 podStartE2EDuration="1.785367258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:32:59.784013264 +0000 UTC m=+1960.801034360" watchObservedRunningTime="2026-07-29 19:32:59.785367258 +0000 UTC m=+1960.802388353" Jul 29 19:33:00.264855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:00.264818 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:00.270225 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:00.270197 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:00.770531 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:00.770490 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:00.771591 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:00.771572 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:03.139168 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.139134 2575 patch_prober.go:28] interesting pod/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 29 19:33:03.139498 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.139202 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 29 19:33:03.371909 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.371886 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:33:03.433949 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.433911 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-token\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.433957 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istiod-ca-cert\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.433974 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-socket\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434020 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-data\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434039 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-credential-socket\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-certs\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-envoy\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434172 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8sgm\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-kube-api-access-k8sgm\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434217 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-podinfo\") pod \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\" (UID: \"91ae4ccf-e770-44dc-8b33-8fc061ff1403\") " Jul 29 19:33:03.434469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434346 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:33:03.434469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434376 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:03.434694 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434624 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.434694 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434643 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.434805 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434740 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:03.434859 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.434809 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:03.435357 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.435323 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-data" (OuterVolumeSpecName: "istio-data") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:03.436460 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.436432 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:03.436684 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.436661 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-token" (OuterVolumeSpecName: "istio-token") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:33:03.436969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.436932 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:33:03.437062 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.436973 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-kube-api-access-k8sgm" (OuterVolumeSpecName: "kube-api-access-k8sgm") pod "91ae4ccf-e770-44dc-8b33-8fc061ff1403" (UID: "91ae4ccf-e770-44dc-8b33-8fc061ff1403"). InnerVolumeSpecName "kube-api-access-k8sgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:33:03.535701 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.535666 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.535701 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.535696 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sgm\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-kube-api-access-k8sgm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.535928 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.535709 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.535928 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.535722 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.535928 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.535734 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.535928 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.535745 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.535928 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.535757 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ae4ccf-e770-44dc-8b33-8fc061ff1403-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:03.789737 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.789647 2575 generic.go:356] "Generic (PLEG): container finished" podID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerID="5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b" exitCode=0 Jul 29 19:33:03.789737 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.789698 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" Jul 29 19:33:03.789920 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.789700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" event={"ID":"91ae4ccf-e770-44dc-8b33-8fc061ff1403","Type":"ContainerDied","Data":"5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b"} Jul 29 19:33:03.789920 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.789822 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9" event={"ID":"91ae4ccf-e770-44dc-8b33-8fc061ff1403","Type":"ContainerDied","Data":"f661b602ae82e0e83a9b984593649c195b0782125a4f7022bfbc6ac7fc202246"} Jul 29 19:33:03.789920 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.789846 2575 scope.go:117] "RemoveContainer" containerID="5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b" Jul 29 19:33:03.799597 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.799548 2575 scope.go:117] "RemoveContainer" containerID="041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53" Jul 29 19:33:03.808099 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.808077 2575 scope.go:117] "RemoveContainer" containerID="5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b" Jul 29 19:33:03.808369 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:33:03.808343 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b\": container with ID starting with 5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b not found: ID does not exist" containerID="5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b" Jul 29 19:33:03.808482 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.808381 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b"} err="failed to get container status \"5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b\": rpc error: code = NotFound desc = could not find container \"5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b\": container with ID starting with 5b3881e602c461cc7ccd883ae75faf9bfeb36664c8accb44fcae30f72717130b not found: ID does not exist" Jul 29 19:33:03.808482 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.808404 2575 scope.go:117] "RemoveContainer" containerID="041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53" Jul 29 19:33:03.808704 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:33:03.808684 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53\": container with ID starting with 041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53 not found: ID does not exist" containerID="041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53" Jul 29 19:33:03.808760 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.808709 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53"} err="failed to get container status \"041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53\": rpc error: code = NotFound desc = could not find container \"041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53\": container with ID starting with 041c0982f97ae9b4c1bb1dbecfc0141c2ae3c82b4275128165faf30e7e2bde53 not found: ID does not exist" Jul 29 19:33:03.810258 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.810238 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9"] Jul 29 19:33:03.815830 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:03.815806 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e80635b4-gw-openshift-default-5f55976948-dw9q9"] Jul 29 19:33:05.549947 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:05.549915 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" path="/var/lib/kubelet/pods/91ae4ccf-e770-44dc-8b33-8fc061ff1403/volumes" Jul 29 19:33:15.771927 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:15.771898 2575 patch_prober.go:28] interesting pod/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:33:15.772251 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:15.771974 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 19:33:15.867994 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:15.867961 2575 generic.go:356] "Generic (PLEG): container finished" podID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerID="af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb" exitCode=0 Jul 29 19:33:15.868171 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:15.868037 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" event={"ID":"75b88b09-ec86-4812-986d-f98af3fd2a1e","Type":"ContainerDied","Data":"af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb"} Jul 29 19:33:15.868479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:15.868462 2575 scope.go:117] "RemoveContainer" containerID="af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb" Jul 29 19:33:15.869603 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:15.869575 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:33:15.869693 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:15.869647 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:33:15.869693 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:15.869680 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:33:16.875429 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:16.875393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" event={"ID":"75b88b09-ec86-4812-986d-f98af3fd2a1e","Type":"ContainerStarted","Data":"7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9"} Jul 29 19:33:17.264189 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:17.264093 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:17.264366 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:17.264308 2575 patch_prober.go:28] interesting pod/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:33:17.264439 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:17.264360 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 19:33:18.264939 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:18.264900 2575 patch_prober.go:28] interesting pod/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:33:18.265366 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:18.264973 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 19:33:19.264636 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:19.264597 2575 patch_prober.go:28] interesting pod/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:33:19.264835 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:19.264664 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 19:33:20.264524 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:20.264467 2575 patch_prober.go:28] interesting pod/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:33:20.264936 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:20.264558 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 29 19:33:21.267959 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:21.267931 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:21.268399 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:21.268319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:21.269073 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:21.269054 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:43.166584 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.166546 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6"] Jul 29 19:33:43.167023 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.166768 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" containerID="cri-o://7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9" gracePeriod=30 Jul 29 19:33:43.899225 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.899194 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk"] Jul 29 19:33:43.899645 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.899631 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" Jul 29 19:33:43.899717 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.899646 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" Jul 29 19:33:43.899717 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.899667 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" Jul 29 19:33:43.899717 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.899673 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" Jul 29 19:33:43.899812 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.899745 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" Jul 29 19:33:43.899812 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.899757 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ae4ccf-e770-44dc-8b33-8fc061ff1403" containerName="istio-proxy" Jul 29 19:33:43.901895 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.901878 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:43.904567 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.904505 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3a338fad-openshift-default-dockercfg-sfclh\"" Jul 29 19:33:43.918293 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:43.918266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk"] Jul 29 19:33:44.031851 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.031815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-envoy\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.032027 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.031866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9739806c-501a-4bac-a093-8bfd41371587-istio-podinfo\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.032027 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.031925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-data\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.032027 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.031970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-istio-token\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.032202 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.032049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-certs\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.032202 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.032095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9739806c-501a-4bac-a093-8bfd41371587-istiod-ca-cert\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.032202 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.032134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-socket\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.032350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.032229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8sng\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-kube-api-access-x8sng\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.032350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.032278 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-credential-socket\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.133708 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.133672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8sng\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-kube-api-access-x8sng\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.133882 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.133725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-credential-socket\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.133882 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.133755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-envoy\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.133882 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.133800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9739806c-501a-4bac-a093-8bfd41371587-istio-podinfo\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134040 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.133931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-data\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134040 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.133975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-istio-token\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134123 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.134048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-certs\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134123 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.134080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9739806c-501a-4bac-a093-8bfd41371587-istiod-ca-cert\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134123 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.134112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-socket\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134294 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.134262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-credential-socket\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134477 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.134308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-data\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134477 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.134458 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-certs\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134687 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.134559 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-socket\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.134788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.134767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9739806c-501a-4bac-a093-8bfd41371587-istiod-ca-cert\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.136053 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.136033 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-envoy\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.136409 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.136392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9739806c-501a-4bac-a093-8bfd41371587-istio-podinfo\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.143444 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.143425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8sng\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-kube-api-access-x8sng\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.143709 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.143689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-istio-token\") pod \"e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.219097 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.219017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:44.366816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.366787 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk"] Jul 29 19:33:44.368199 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:33:44.368173 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9739806c_501a_4bac_a093_8bfd41371587.slice/crio-1ba8c3128b74a606999ce81f7325e4f06a929069f5f1e0bf069e35362e13a51a WatchSource:0}: Error finding container 1ba8c3128b74a606999ce81f7325e4f06a929069f5f1e0bf069e35362e13a51a: Status 404 returned error can't find the container with id 1ba8c3128b74a606999ce81f7325e4f06a929069f5f1e0bf069e35362e13a51a Jul 29 19:33:44.370697 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.370666 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:33:44.370792 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.370728 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:33:44.370792 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:44.370758 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:33:45.045432 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:45.045397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" event={"ID":"9739806c-501a-4bac-a093-8bfd41371587","Type":"ContainerStarted","Data":"6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7"} Jul 29 19:33:45.045614 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:45.045448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" event={"ID":"9739806c-501a-4bac-a093-8bfd41371587","Type":"ContainerStarted","Data":"1ba8c3128b74a606999ce81f7325e4f06a929069f5f1e0bf069e35362e13a51a"} Jul 29 19:33:45.068240 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:45.068191 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" podStartSLOduration=2.068172017 podStartE2EDuration="2.068172017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:33:45.064831015 +0000 UTC m=+2006.081852109" watchObservedRunningTime="2026-07-29 19:33:45.068172017 +0000 UTC m=+2006.085193162" Jul 29 19:33:45.220050 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:45.220015 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:45.224939 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:45.224918 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:46.051667 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:46.051634 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:46.052494 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:46.052476 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:33:48.430363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.430334 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:48.580058 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.579975 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b88b09-ec86-4812-986d-f98af3fd2a1e-istiod-ca-cert\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580204 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580059 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-data\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580204 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580081 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-token\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580204 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580123 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-socket\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580204 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580147 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-envoy\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580204 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580203 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-podinfo\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580229 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-credential-socket\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580260 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w82n2\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-kube-api-access-w82n2\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580446 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580361 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-certs\") pod \"75b88b09-ec86-4812-986d-f98af3fd2a1e\" (UID: \"75b88b09-ec86-4812-986d-f98af3fd2a1e\") " Jul 29 19:33:48.580634 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580454 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b88b09-ec86-4812-986d-f98af3fd2a1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:33:48.580634 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580472 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:48.580751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580670 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:48.580751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580683 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:48.580851 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580766 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:48.580851 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580786 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:48.580851 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580800 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b88b09-ec86-4812-986d-f98af3fd2a1e-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:48.580851 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580813 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:48.581050 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.580990 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:48.582453 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.582427 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:33:48.582655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.582483 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:33:48.582655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.582603 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-kube-api-access-w82n2" (OuterVolumeSpecName: "kube-api-access-w82n2") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "kube-api-access-w82n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:33:48.582655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.582626 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75b88b09-ec86-4812-986d-f98af3fd2a1e" (UID: "75b88b09-ec86-4812-986d-f98af3fd2a1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:33:48.681768 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.681719 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:48.681768 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.681763 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w82n2\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-kube-api-access-w82n2\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:48.681768 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.681778 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:48.681985 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.681790 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:48.681985 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:48.681802 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b88b09-ec86-4812-986d-f98af3fd2a1e-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:33:49.073740 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.073703 2575 generic.go:356] "Generic (PLEG): container finished" podID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerID="7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9" exitCode=0 Jul 29 19:33:49.073915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.073746 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" Jul 29 19:33:49.073915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.073752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" event={"ID":"75b88b09-ec86-4812-986d-f98af3fd2a1e","Type":"ContainerDied","Data":"7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9"} Jul 29 19:33:49.073915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.073808 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6" event={"ID":"75b88b09-ec86-4812-986d-f98af3fd2a1e","Type":"ContainerDied","Data":"e7c2a827dab9175aea76666bdb31209db9f824149ff45538e30f8336302a1915"} Jul 29 19:33:49.073915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.073833 2575 scope.go:117] "RemoveContainer" containerID="7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9" Jul 29 19:33:49.083985 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.083965 2575 scope.go:117] "RemoveContainer" containerID="af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb" Jul 29 19:33:49.092319 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.092301 2575 scope.go:117] "RemoveContainer" containerID="7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9" Jul 29 19:33:49.092568 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:33:49.092547 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9\": container with ID starting with 7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9 not found: ID does not exist" containerID="7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9" Jul 29 19:33:49.092622 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.092577 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9"} err="failed to get container status \"7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9\": rpc error: code = NotFound desc = could not find container \"7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9\": container with ID starting with 7175eb9d515392291fafbb281d1eba6864120eb42ea25e964a2641680bf29af9 not found: ID does not exist" Jul 29 19:33:49.092622 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.092594 2575 scope.go:117] "RemoveContainer" containerID="af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb" Jul 29 19:33:49.092833 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:33:49.092815 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb\": container with ID starting with af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb not found: ID does not exist" containerID="af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb" Jul 29 19:33:49.092879 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.092839 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb"} err="failed to get container status \"af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb\": rpc error: code = NotFound desc = could not find container \"af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb\": container with ID starting with af9575472a7806ddefd4bf3ef26e79d1c01a533a50967c5dc29ad83a9c2e1adb not found: ID does not exist" Jul 29 19:33:49.098737 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.098713 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6"] Jul 29 19:33:49.102603 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.102581 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6d42a5b6-openshift-default-5bf8d994fb-rfqz6"] Jul 29 19:33:49.551703 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:33:49.551674 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" path="/var/lib/kubelet/pods/75b88b09-ec86-4812-986d-f98af3fd2a1e/volumes" Jul 29 19:34:00.145188 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:00.145145 2575 generic.go:356] "Generic (PLEG): container finished" podID="9739806c-501a-4bac-a093-8bfd41371587" containerID="6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7" exitCode=0 Jul 29 19:34:00.145662 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:00.145222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" event={"ID":"9739806c-501a-4bac-a093-8bfd41371587","Type":"ContainerDied","Data":"6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7"} Jul 29 19:34:00.145837 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:00.145819 2575 scope.go:117] "RemoveContainer" containerID="6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7" Jul 29 19:34:00.147060 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:00.147029 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:34:00.147175 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:00.147097 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:34:00.147175 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:00.147141 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:34:01.051919 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:01.051872 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:34:01.159739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:01.159697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" event={"ID":"9739806c-501a-4bac-a093-8bfd41371587","Type":"ContainerStarted","Data":"62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8"} Jul 29 19:34:01.219912 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:01.219879 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:34:01.220831 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:01.220797 2575 patch_prober.go:28] interesting pod/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 29 19:34:01.220954 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:01.220845 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 29 19:34:02.219603 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:02.219564 2575 patch_prober.go:28] interesting pod/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 29 19:34:02.220023 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:02.219638 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 29 19:34:03.219812 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:03.219782 2575 patch_prober.go:28] interesting pod/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 29 19:34:03.220188 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:03.219834 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 29 19:34:04.219659 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:04.219622 2575 patch_prober.go:28] interesting pod/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 29 19:34:04.219834 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:04.219675 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 29 19:34:05.223888 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:05.223859 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:34:05.224266 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:05.224121 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:34:05.224684 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:05.224668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:34:17.510723 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:17.510676 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk"] Jul 29 19:34:17.511239 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:17.510942 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" containerID="cri-o://62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8" gracePeriod=30 Jul 29 19:34:20.226422 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:20.226378 2575 patch_prober.go:28] interesting pod/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 29 19:34:20.226837 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:20.226455 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:34:22.779439 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.779386 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:34:22.924137 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924109 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9739806c-501a-4bac-a093-8bfd41371587-istio-podinfo\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924149 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-socket\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924168 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-credential-socket\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924193 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9739806c-501a-4bac-a093-8bfd41371587-istiod-ca-cert\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924222 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-data\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924251 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-certs\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924325 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924279 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8sng\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-kube-api-access-x8sng\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924722 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924338 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-istio-token\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924722 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924362 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-envoy\") pod \"9739806c-501a-4bac-a093-8bfd41371587\" (UID: \"9739806c-501a-4bac-a093-8bfd41371587\") " Jul 29 19:34:22.924722 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924462 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:34:22.924722 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924471 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:34:22.924722 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924688 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9739806c-501a-4bac-a093-8bfd41371587-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:34:22.924942 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924800 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:22.924942 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924817 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:22.924942 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924831 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9739806c-501a-4bac-a093-8bfd41371587-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:22.924942 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.924838 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:34:22.925367 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.925324 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-data" (OuterVolumeSpecName: "istio-data") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:34:22.926526 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.926491 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-istio-token" (OuterVolumeSpecName: "istio-token") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:34:22.926655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.926634 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:34:22.926789 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.926774 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9739806c-501a-4bac-a093-8bfd41371587-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:34:22.926869 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:22.926850 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-kube-api-access-x8sng" (OuterVolumeSpecName: "kube-api-access-x8sng") pod "9739806c-501a-4bac-a093-8bfd41371587" (UID: "9739806c-501a-4bac-a093-8bfd41371587"). InnerVolumeSpecName "kube-api-access-x8sng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:34:23.026037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.026008 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:23.026037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.026033 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9739806c-501a-4bac-a093-8bfd41371587-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:23.026037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.026042 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:23.026263 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.026050 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9739806c-501a-4bac-a093-8bfd41371587-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:23.026263 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.026059 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8sng\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-kube-api-access-x8sng\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:23.026263 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.026070 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9739806c-501a-4bac-a093-8bfd41371587-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:34:23.288564 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.288529 2575 generic.go:356] "Generic (PLEG): container finished" podID="9739806c-501a-4bac-a093-8bfd41371587" containerID="62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8" exitCode=0 Jul 29 19:34:23.288769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.288586 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" Jul 29 19:34:23.288769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.288606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" event={"ID":"9739806c-501a-4bac-a093-8bfd41371587","Type":"ContainerDied","Data":"62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8"} Jul 29 19:34:23.288769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.288665 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk" event={"ID":"9739806c-501a-4bac-a093-8bfd41371587","Type":"ContainerDied","Data":"1ba8c3128b74a606999ce81f7325e4f06a929069f5f1e0bf069e35362e13a51a"} Jul 29 19:34:23.288769 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.288692 2575 scope.go:117] "RemoveContainer" containerID="62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8" Jul 29 19:34:23.305724 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.305702 2575 scope.go:117] "RemoveContainer" containerID="6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7" Jul 29 19:34:23.314044 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.314015 2575 scope.go:117] "RemoveContainer" containerID="62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8" Jul 29 19:34:23.314268 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:34:23.314245 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8\": container with ID starting with 62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8 not found: ID does not exist" containerID="62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8" Jul 29 19:34:23.314416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.314276 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8"} err="failed to get container status \"62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8\": rpc error: code = NotFound desc = could not find container \"62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8\": container with ID starting with 62e135490cf40c3184e6852487a84516a075160db05afebcc793dc4886f63ca8 not found: ID does not exist" Jul 29 19:34:23.314416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.314294 2575 scope.go:117] "RemoveContainer" containerID="6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7" Jul 29 19:34:23.314574 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:34:23.314546 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7\": container with ID starting with 6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7 not found: ID does not exist" containerID="6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7" Jul 29 19:34:23.314637 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.314583 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7"} err="failed to get container status \"6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7\": rpc error: code = NotFound desc = could not find container \"6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7\": container with ID starting with 6ff076a3be2ae18dc7db531d317c771df755785041330a27a0c165dd91ad3cf7 not found: ID does not exist" Jul 29 19:34:23.317185 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.317159 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk"] Jul 29 19:34:23.319651 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.319627 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3a338fad-openshift-default-6cc4869868-lfclk"] Jul 29 19:34:23.550830 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:34:23.550737 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9739806c-501a-4bac-a093-8bfd41371587" path="/var/lib/kubelet/pods/9739806c-501a-4bac-a093-8bfd41371587/volumes" Jul 29 19:36:07.108788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.108758 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t"] Jul 29 19:36:07.109257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.109156 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" Jul 29 19:36:07.109257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.109169 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" Jul 29 19:36:07.109257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.109181 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" Jul 29 19:36:07.109257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.109186 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" Jul 29 19:36:07.109257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.109198 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" Jul 29 19:36:07.109257 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.109204 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" Jul 29 19:36:07.109466 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.109275 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" Jul 29 19:36:07.109466 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.109283 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" Jul 29 19:36:07.111590 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.111566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.114354 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.114335 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7beba8e5-openshift-default-dockercfg-bdz4h\"" Jul 29 19:36:07.139772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.139747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t"] Jul 29 19:36:07.294070 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-credential-socket\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.294070 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-socket\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.294271 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-istio-token\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.294271 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1d9c39-3daa-463c-af8c-323933288041-istio-podinfo\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.294271 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-certs\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.294271 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-envoy\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.294271 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1d9c39-3daa-463c-af8c-323933288041-istiod-ca-cert\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.294271 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-data\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.294271 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.294258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46sn\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-kube-api-access-x46sn\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.395784 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-credential-socket\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.395968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-socket\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.395968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-istio-token\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.395968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1d9c39-3daa-463c-af8c-323933288041-istio-podinfo\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.395968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-certs\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.395968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-envoy\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.395968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1d9c39-3daa-463c-af8c-323933288041-istiod-ca-cert\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.395968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-data\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.396363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.395978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46sn\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-kube-api-access-x46sn\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.396363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.396225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-socket\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.396363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.396337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-certs\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.396555 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.396436 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-data\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.396555 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.396452 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-credential-socket\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.396763 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.396740 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1d9c39-3daa-463c-af8c-323933288041-istiod-ca-cert\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.398174 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.398153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-envoy\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.398373 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.398357 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1d9c39-3daa-463c-af8c-323933288041-istio-podinfo\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.404455 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.404436 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-istio-token\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.404618 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.404596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46sn\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-kube-api-access-x46sn\") pod \"e2e-mt-7beba8e5-openshift-default-6749858548-fk65t\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.422435 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.422417 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:07.569541 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.566569 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t"] Jul 29 19:36:07.575220 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.575181 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:36:07.575321 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.575253 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:36:07.575321 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.575280 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:36:07.922127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.922079 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" event={"ID":"de1d9c39-3daa-463c-af8c-323933288041","Type":"ContainerStarted","Data":"9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad"} Jul 29 19:36:07.922332 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.922157 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" event={"ID":"de1d9c39-3daa-463c-af8c-323933288041","Type":"ContainerStarted","Data":"ffdc917f6ec59a16acf7503b7ae19240e50779f1e97629a4042bd741476d2c29"} Jul 29 19:36:07.945831 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:07.945752 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" podStartSLOduration=0.945736342 podStartE2EDuration="945.736342ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:36:07.943771027 +0000 UTC m=+2148.960792124" watchObservedRunningTime="2026-07-29 19:36:07.945736342 +0000 UTC m=+2148.962757438" Jul 29 19:36:08.423690 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:08.423654 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:08.428457 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:08.428435 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:08.928758 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:08.928721 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:08.929662 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:08.929640 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:23.017065 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:23.017034 2575 generic.go:356] "Generic (PLEG): container finished" podID="de1d9c39-3daa-463c-af8c-323933288041" containerID="9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad" exitCode=0 Jul 29 19:36:23.017616 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:23.017111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" event={"ID":"de1d9c39-3daa-463c-af8c-323933288041","Type":"ContainerDied","Data":"9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad"} Jul 29 19:36:23.017689 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:23.017652 2575 scope.go:117] "RemoveContainer" containerID="9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad" Jul 29 19:36:23.018739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:23.018706 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:36:23.018837 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:23.018790 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:36:23.018837 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:23.018831 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:36:23.928763 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:23.928729 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:24.024997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:24.024955 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" event={"ID":"de1d9c39-3daa-463c-af8c-323933288041","Type":"ContainerStarted","Data":"c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486"} Jul 29 19:36:24.423068 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:24.423038 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:24.424237 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:24.424208 2575 patch_prober.go:28] interesting pod/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 19:36:24.424335 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:24.424314 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 19:36:25.423244 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:25.423198 2575 patch_prober.go:28] interesting pod/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 19:36:25.423683 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:25.423264 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 19:36:26.423466 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:26.423431 2575 patch_prober.go:28] interesting pod/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 19:36:26.423851 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:26.423494 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 19:36:27.423176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:27.423139 2575 patch_prober.go:28] interesting pod/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 29 19:36:27.423350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:27.423192 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 29 19:36:28.427100 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:28.427062 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:28.427465 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:28.427414 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:28.428004 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:28.427984 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:36:59.922470 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:59.922436 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t"] Jul 29 19:36:59.923043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:36:59.922648 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" containerID="cri-o://c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486" gracePeriod=30 Jul 29 19:37:05.185383 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.185320 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:37:05.237231 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237201 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-credential-socket\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237236 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-data\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237271 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x46sn\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-kube-api-access-x46sn\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237294 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1d9c39-3daa-463c-af8c-323933288041-istiod-ca-cert\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237335 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-istio-token\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237357 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-certs\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237400 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1d9c39-3daa-463c-af8c-323933288041-istio-podinfo\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237425 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-envoy\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237451 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-socket\") pod \"de1d9c39-3daa-463c-af8c-323933288041\" (UID: \"de1d9c39-3daa-463c-af8c-323933288041\") " Jul 29 19:37:05.237739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237569 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:37:05.237896 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237803 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.237896 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237814 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1d9c39-3daa-463c-af8c-323933288041-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:37:05.237896 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.237816 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:37:05.238045 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.238012 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:37:05.238174 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.238151 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-data" (OuterVolumeSpecName: "istio-data") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:37:05.239587 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.239562 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:37:05.239681 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.239595 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-kube-api-access-x46sn" (OuterVolumeSpecName: "kube-api-access-x46sn") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "kube-api-access-x46sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:37:05.240043 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.240020 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-istio-token" (OuterVolumeSpecName: "istio-token") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:37:05.240121 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.240020 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de1d9c39-3daa-463c-af8c-323933288041-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de1d9c39-3daa-463c-af8c-323933288041" (UID: "de1d9c39-3daa-463c-af8c-323933288041"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:37:05.262649 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.262621 2575 generic.go:356] "Generic (PLEG): container finished" podID="de1d9c39-3daa-463c-af8c-323933288041" containerID="c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486" exitCode=0 Jul 29 19:37:05.262744 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.262672 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" Jul 29 19:37:05.262744 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.262699 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" event={"ID":"de1d9c39-3daa-463c-af8c-323933288041","Type":"ContainerDied","Data":"c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486"} Jul 29 19:37:05.262825 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.262750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t" event={"ID":"de1d9c39-3daa-463c-af8c-323933288041","Type":"ContainerDied","Data":"ffdc917f6ec59a16acf7503b7ae19240e50779f1e97629a4042bd741476d2c29"} Jul 29 19:37:05.262825 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.262773 2575 scope.go:117] "RemoveContainer" containerID="c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486" Jul 29 19:37:05.272645 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.272627 2575 scope.go:117] "RemoveContainer" containerID="9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad" Jul 29 19:37:05.280922 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.280898 2575 scope.go:117] "RemoveContainer" containerID="c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486" Jul 29 19:37:05.281191 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:37:05.281174 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486\": container with ID starting with c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486 not found: ID does not exist" containerID="c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486" Jul 29 19:37:05.281235 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.281202 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486"} err="failed to get container status \"c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486\": rpc error: code = NotFound desc = could not find container \"c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486\": container with ID starting with c8cbedc8ca4d19e9f533bfa67d88e94a449fd8908017b71d808c94eeacce5486 not found: ID does not exist" Jul 29 19:37:05.281235 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.281219 2575 scope.go:117] "RemoveContainer" containerID="9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad" Jul 29 19:37:05.281473 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:37:05.281456 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad\": container with ID starting with 9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad not found: ID does not exist" containerID="9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad" Jul 29 19:37:05.281550 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.281483 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad"} err="failed to get container status \"9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad\": rpc error: code = NotFound desc = could not find container \"9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad\": container with ID starting with 9afed424047afa404435246df3dcfd7b61374b099d975686614cd6ab10eb49ad not found: ID does not exist" Jul 29 19:37:05.287506 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.287484 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t"] Jul 29 19:37:05.291957 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.291937 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7beba8e5-openshift-default-6749858548-fk65t"] Jul 29 19:37:05.338545 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.338479 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.338545 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.338510 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46sn\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-kube-api-access-x46sn\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.338751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.338556 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1d9c39-3daa-463c-af8c-323933288041-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.338751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.338569 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1d9c39-3daa-463c-af8c-323933288041-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.338751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.338580 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.338751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.338590 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1d9c39-3daa-463c-af8c-323933288041-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.338751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.338600 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.338751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.338610 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1d9c39-3daa-463c-af8c-323933288041-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:37:05.549342 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:05.549257 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1d9c39-3daa-463c-af8c-323933288041" path="/var/lib/kubelet/pods/de1d9c39-3daa-463c-af8c-323933288041/volumes" Jul 29 19:37:19.668578 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.668545 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99"] Jul 29 19:37:19.669175 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669156 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" Jul 29 19:37:19.669289 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669177 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" Jul 29 19:37:19.669289 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669207 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" Jul 29 19:37:19.669289 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669216 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" Jul 29 19:37:19.669289 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669228 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" Jul 29 19:37:19.669289 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669237 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" Jul 29 19:37:19.669555 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669343 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" Jul 29 19:37:19.669555 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669358 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1d9c39-3daa-463c-af8c-323933288041" containerName="istio-proxy" Jul 29 19:37:19.669555 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669371 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9739806c-501a-4bac-a093-8bfd41371587" containerName="istio-proxy" Jul 29 19:37:19.669555 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.669384 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b88b09-ec86-4812-986d-f98af3fd2a1e" containerName="istio-proxy" Jul 29 19:37:19.681060 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.681037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.683839 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.683654 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-77b3a01f-openshift-default-dockercfg-kpg6b\"" Jul 29 19:37:19.685572 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.685545 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99"] Jul 29 19:37:19.770688 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istiod-ca-cert\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.770688 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-certs\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.770889 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-envoy\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.770889 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-data\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.770889 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-credential-socket\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.770889 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tjzp\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-kube-api-access-4tjzp\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.770889 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-podinfo\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.770889 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-token\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.771152 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.770964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-socket\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872232 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istiod-ca-cert\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872232 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-certs\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872443 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872275 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-envoy\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872443 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-data\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872443 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-credential-socket\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872634 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tjzp\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-kube-api-access-4tjzp\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872634 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-podinfo\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-token\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-socket\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-certs\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872890 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872775 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-credential-socket\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.872890 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.872836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-data\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.873108 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.873083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-socket\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.873215 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.873085 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istiod-ca-cert\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.874840 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.874821 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-envoy\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.875049 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.875029 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-podinfo\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.880905 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.880884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-token\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.881201 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.881180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tjzp\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-kube-api-access-4tjzp\") pod \"e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:19.993179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:19.993094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:20.141147 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:20.141121 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99"] Jul 29 19:37:20.143424 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:37:20.143393 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dd4005_a8c2_4bcb_8fba_ed662aae32de.slice/crio-3a335b9eb85f93f70ffcde6702e2bf41d418c76d9d06ddb11cb716388b39d484 WatchSource:0}: Error finding container 3a335b9eb85f93f70ffcde6702e2bf41d418c76d9d06ddb11cb716388b39d484: Status 404 returned error can't find the container with id 3a335b9eb85f93f70ffcde6702e2bf41d418c76d9d06ddb11cb716388b39d484 Jul 29 19:37:20.145476 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:20.145441 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:37:20.145568 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:20.145508 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:37:20.145568 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:20.145553 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:37:20.349144 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:20.349040 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" event={"ID":"31dd4005-a8c2-4bcb-8fba-ed662aae32de","Type":"ContainerStarted","Data":"fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee"} Jul 29 19:37:20.349144 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:20.349108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" event={"ID":"31dd4005-a8c2-4bcb-8fba-ed662aae32de","Type":"ContainerStarted","Data":"3a335b9eb85f93f70ffcde6702e2bf41d418c76d9d06ddb11cb716388b39d484"} Jul 29 19:37:20.373194 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:20.373132 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" podStartSLOduration=1.373116897 podStartE2EDuration="1.373116897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:37:20.370372513 +0000 UTC m=+2221.387393610" watchObservedRunningTime="2026-07-29 19:37:20.373116897 +0000 UTC m=+2221.390137992" Jul 29 19:37:20.993890 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:20.993840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:21.999055 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:21.999025 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:22.359725 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:22.359645 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:22.360786 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:22.360759 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:36.456278 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:36.456243 2575 generic.go:356] "Generic (PLEG): container finished" podID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerID="fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee" exitCode=0 Jul 29 19:37:36.456750 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:36.456302 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" event={"ID":"31dd4005-a8c2-4bcb-8fba-ed662aae32de","Type":"ContainerDied","Data":"fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee"} Jul 29 19:37:36.456750 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:36.456702 2575 scope.go:117] "RemoveContainer" containerID="fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee" Jul 29 19:37:36.457739 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:36.457706 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:37:36.457832 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:36.457775 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:37:36.457832 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:36.457806 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:37:37.360438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:37.360389 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:37.465717 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:37.465678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" event={"ID":"31dd4005-a8c2-4bcb-8fba-ed662aae32de","Type":"ContainerStarted","Data":"26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269"} Jul 29 19:37:37.993433 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:37.993399 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:37.994451 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:37.994416 2575 patch_prober.go:28] interesting pod/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:37:37.994624 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:37.994464 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 29 19:37:38.994039 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:38.994005 2575 patch_prober.go:28] interesting pod/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:37:38.994469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:38.994057 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 29 19:37:39.994109 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:39.994074 2575 patch_prober.go:28] interesting pod/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:37:39.994490 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:39.994140 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 29 19:37:40.993718 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:40.993683 2575 patch_prober.go:28] interesting pod/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:37:40.993895 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:40.993742 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 29 19:37:41.997838 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:41.997757 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:41.998309 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:41.998287 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:37:41.999078 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:37:41.999054 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:39:47.831149 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:47.831116 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99"] Jul 29 19:39:47.831633 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:47.831336 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" containerID="cri-o://26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269" gracePeriod=30 Jul 29 19:39:53.106138 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.106053 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:39:53.143255 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143215 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-socket\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.143448 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143266 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-credential-socket\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.143448 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143303 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tjzp\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-kube-api-access-4tjzp\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.143448 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143361 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-envoy\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.143448 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143413 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-data\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.143707 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143508 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:39:53.143707 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143606 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:39:53.143707 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143650 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-token\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.143868 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143727 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-certs\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.143868 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143755 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-podinfo\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.143868 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.143818 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istiod-ca-cert\") pod \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\" (UID: \"31dd4005-a8c2-4bcb-8fba-ed662aae32de\") " Jul 29 19:39:53.144168 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.144142 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:39:53.144382 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.144186 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.144536 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.144495 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.144673 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.144437 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:39:53.144795 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.144474 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-data" (OuterVolumeSpecName: "istio-data") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:39:53.145884 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.145848 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:39:53.146440 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.146410 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-kube-api-access-4tjzp" (OuterVolumeSpecName: "kube-api-access-4tjzp") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "kube-api-access-4tjzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:39:53.146538 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.146419 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-token" (OuterVolumeSpecName: "istio-token") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:39:53.146743 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.146727 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31dd4005-a8c2-4bcb-8fba-ed662aae32de" (UID: "31dd4005-a8c2-4bcb-8fba-ed662aae32de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:39:53.242936 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.241742 2575 generic.go:356] "Generic (PLEG): container finished" podID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerID="26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269" exitCode=0 Jul 29 19:39:53.242936 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.241863 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" event={"ID":"31dd4005-a8c2-4bcb-8fba-ed662aae32de","Type":"ContainerDied","Data":"26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269"} Jul 29 19:39:53.242936 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.241966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" event={"ID":"31dd4005-a8c2-4bcb-8fba-ed662aae32de","Type":"ContainerDied","Data":"3a335b9eb85f93f70ffcde6702e2bf41d418c76d9d06ddb11cb716388b39d484"} Jul 29 19:39:53.242936 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.241995 2575 scope.go:117] "RemoveContainer" containerID="26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269" Jul 29 19:39:53.242936 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.242160 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99" Jul 29 19:39:53.245918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.245649 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.245918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.245673 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.245918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.245685 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.245918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.245696 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dd4005-a8c2-4bcb-8fba-ed662aae32de-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.245918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.245707 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.245918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.245716 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dd4005-a8c2-4bcb-8fba-ed662aae32de-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.245918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.245727 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tjzp\" (UniqueName: \"kubernetes.io/projected/31dd4005-a8c2-4bcb-8fba-ed662aae32de-kube-api-access-4tjzp\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:39:53.268897 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.268645 2575 scope.go:117] "RemoveContainer" containerID="fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee" Jul 29 19:39:53.279306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.279284 2575 scope.go:117] "RemoveContainer" containerID="26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269" Jul 29 19:39:53.279626 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:39:53.279606 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269\": container with ID starting with 26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269 not found: ID does not exist" containerID="26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269" Jul 29 19:39:53.279697 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.279637 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269"} err="failed to get container status \"26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269\": rpc error: code = NotFound desc = could not find container \"26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269\": container with ID starting with 26e0254aef4ed1a2fd41ea335f6574b321141395e4f08174b460a61184187269 not found: ID does not exist" Jul 29 19:39:53.279697 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.279658 2575 scope.go:117] "RemoveContainer" containerID="fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee" Jul 29 19:39:53.279952 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:39:53.279933 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee\": container with ID starting with fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee not found: ID does not exist" containerID="fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee" Jul 29 19:39:53.280011 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.279962 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee"} err="failed to get container status \"fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee\": rpc error: code = NotFound desc = could not find container \"fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee\": container with ID starting with fab30375b518c6ae88c05d21dd331b38967a86dc3464929b692e5d8ab0db60ee not found: ID does not exist" Jul 29 19:39:53.301472 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.301433 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99"] Jul 29 19:39:53.305381 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.305356 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-77b3a01f-openshift-default-6b99c95b8-7ns99"] Jul 29 19:39:53.549893 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:39:53.549860 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" path="/var/lib/kubelet/pods/31dd4005-a8c2-4bcb-8fba-ed662aae32de/volumes" Jul 29 19:40:44.064944 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.064862 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c"] Jul 29 19:40:44.065402 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.065322 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" Jul 29 19:40:44.065402 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.065335 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" Jul 29 19:40:44.065402 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.065345 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" Jul 29 19:40:44.065402 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.065350 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" Jul 29 19:40:44.065564 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.065425 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" Jul 29 19:40:44.065564 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.065433 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31dd4005-a8c2-4bcb-8fba-ed662aae32de" containerName="istio-proxy" Jul 29 19:40:44.068664 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.068649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.071681 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.071654 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-909676-openshift-default-dockercfg-856p8\"" Jul 29 19:40:44.082152 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.082128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c"] Jul 29 19:40:44.134941 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.134910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-socket\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.134941 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.134943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee7e1a9-4076-42b5-baa5-61ce51014813-istiod-ca-cert\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.135119 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.134969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-certs\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.135119 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.135043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-credential-socket\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.135119 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.135095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-envoy\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.135222 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.135145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-podinfo\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.135222 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.135162 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-token\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.135293 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.135221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-data\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.135293 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.135253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjq6k\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-kube-api-access-gjq6k\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236381 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-certs\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-credential-socket\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-envoy\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-podinfo\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-token\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-data\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236899 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjq6k\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-kube-api-access-gjq6k\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236899 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-socket\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.236899 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.236697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee7e1a9-4076-42b5-baa5-61ce51014813-istiod-ca-cert\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.237534 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.237492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee7e1a9-4076-42b5-baa5-61ce51014813-istiod-ca-cert\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.237711 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.237509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-certs\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.237805 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.237753 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-data\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.238136 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.238118 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-socket\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.238707 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.238686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-credential-socket\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.240455 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.240435 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-envoy\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.240587 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.240565 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-podinfo\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.247411 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.247386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjq6k\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-kube-api-access-gjq6k\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.248589 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.248571 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-token\") pod \"e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.379094 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.379057 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:44.526002 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.525971 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c"] Jul 29 19:40:44.526860 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:40:44.526829 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee7e1a9_4076_42b5_baa5_61ce51014813.slice/crio-49a67029d7f5fd053e66687ecb822b4b21209f89d82143fbe8019333d8e81d24 WatchSource:0}: Error finding container 49a67029d7f5fd053e66687ecb822b4b21209f89d82143fbe8019333d8e81d24: Status 404 returned error can't find the container with id 49a67029d7f5fd053e66687ecb822b4b21209f89d82143fbe8019333d8e81d24 Jul 29 19:40:44.529107 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.529074 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:40:44.529197 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.529146 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:40:44.529197 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.529175 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:40:44.539591 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:44.539552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" event={"ID":"6ee7e1a9-4076-42b5-baa5-61ce51014813","Type":"ContainerStarted","Data":"49a67029d7f5fd053e66687ecb822b4b21209f89d82143fbe8019333d8e81d24"} Jul 29 19:40:45.549852 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:45.549819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" event={"ID":"6ee7e1a9-4076-42b5-baa5-61ce51014813","Type":"ContainerStarted","Data":"1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf"} Jul 29 19:40:45.570233 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:45.570181 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" podStartSLOduration=1.5701655410000002 podStartE2EDuration="1.570165541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:45.568580014 +0000 UTC m=+2426.585601111" watchObservedRunningTime="2026-07-29 19:40:45.570165541 +0000 UTC m=+2426.587186638" Jul 29 19:40:46.380230 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:46.380190 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:46.384981 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:46.384954 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:46.553535 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:46.553487 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:40:46.554566 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:40:46.554549 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:41:01.554915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:01.554880 2575 patch_prober.go:28] interesting pod/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 29 19:41:01.555341 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:01.554949 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 29 19:41:01.643057 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:01.643022 2575 generic.go:356] "Generic (PLEG): container finished" podID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerID="1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf" exitCode=0 Jul 29 19:41:01.643221 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:01.643095 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" event={"ID":"6ee7e1a9-4076-42b5-baa5-61ce51014813","Type":"ContainerDied","Data":"1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf"} Jul 29 19:41:01.643552 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:01.643535 2575 scope.go:117] "RemoveContainer" containerID="1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf" Jul 29 19:41:01.644751 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:01.644723 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:01.644815 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:01.644792 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:01.644863 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:01.644823 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:02.655378 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:02.655342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" event={"ID":"6ee7e1a9-4076-42b5-baa5-61ce51014813","Type":"ContainerStarted","Data":"060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed"} Jul 29 19:41:03.379614 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:03.379574 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:41:03.379824 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:03.379800 2575 patch_prober.go:28] interesting pod/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 29 19:41:03.379879 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:03.379862 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 29 19:41:04.380046 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:04.380007 2575 patch_prober.go:28] interesting pod/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 29 19:41:04.380447 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:04.380063 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 29 19:41:05.380288 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:05.380252 2575 patch_prober.go:28] interesting pod/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 29 19:41:05.380720 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:05.380309 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 29 19:41:06.380360 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:06.380324 2575 patch_prober.go:28] interesting pod/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 29 19:41:06.380774 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:06.380382 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 29 19:41:07.383504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:07.383474 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:41:07.383940 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:07.383803 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:41:07.384671 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:07.384652 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:41:13.028424 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.028393 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt"] Jul 29 19:41:13.032652 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.032632 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.035510 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.035489 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b3bf1c-openshift-default-dockercfg-5299l\"" Jul 29 19:41:13.051895 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.051869 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt"] Jul 29 19:41:13.128274 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-credential-socket\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.128274 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92v9v\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-kube-api-access-92v9v\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.128504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128309 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-socket\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.128504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-podinfo\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.128504 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-data\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.128684 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-envoy\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.128684 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-token\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.128771 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-certs\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.128771 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.128758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istiod-ca-cert\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.229601 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istiod-ca-cert\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.229801 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-credential-socket\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.229801 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v9v\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-kube-api-access-92v9v\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.229801 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-socket\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.229801 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-podinfo\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.229801 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-data\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.230092 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-envoy\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.230092 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-token\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.230092 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.229908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-certs\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.230207 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.230121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-credential-socket\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.230207 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.230177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-data\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.230394 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.230371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istiod-ca-cert\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.230394 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.230386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-socket\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.230555 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.230398 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-certs\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.232165 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.232134 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-envoy\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.232258 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.232174 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-podinfo\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.238968 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.238943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-token\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.239395 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.239368 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v9v\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-kube-api-access-92v9v\") pod \"e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.343285 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.343206 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:13.489014 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.488989 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt"] Jul 29 19:41:13.490832 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:41:13.490801 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b61dad_7cbe_4ebd_b497_a17160dd779c.slice/crio-e5005a49ea75dc8dc2284695d345000727b6666c411ccd8e956a87f11abc14ed WatchSource:0}: Error finding container e5005a49ea75dc8dc2284695d345000727b6666c411ccd8e956a87f11abc14ed: Status 404 returned error can't find the container with id e5005a49ea75dc8dc2284695d345000727b6666c411ccd8e956a87f11abc14ed Jul 29 19:41:13.493060 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.493028 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:13.493147 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.493097 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:13.493192 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.493149 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:13.722952 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.722917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" event={"ID":"e3b61dad-7cbe-4ebd-b497-a17160dd779c","Type":"ContainerStarted","Data":"01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9"} Jul 29 19:41:13.723166 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.722968 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" event={"ID":"e3b61dad-7cbe-4ebd-b497-a17160dd779c","Type":"ContainerStarted","Data":"e5005a49ea75dc8dc2284695d345000727b6666c411ccd8e956a87f11abc14ed"} Jul 29 19:41:13.744251 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:13.744186 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" podStartSLOduration=0.744165374 podStartE2EDuration="744.165374ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:41:13.742474547 +0000 UTC m=+2454.759495645" watchObservedRunningTime="2026-07-29 19:41:13.744165374 +0000 UTC m=+2454.761186469" Jul 29 19:41:14.343813 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:14.343768 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:14.348479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:14.348454 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:14.728882 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:14.728852 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:14.729744 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:14.729721 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:28.825366 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:28.825310 2575 generic.go:356] "Generic (PLEG): container finished" podID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerID="01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9" exitCode=0 Jul 29 19:41:28.825809 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:28.825396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" event={"ID":"e3b61dad-7cbe-4ebd-b497-a17160dd779c","Type":"ContainerDied","Data":"01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9"} Jul 29 19:41:28.825971 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:28.825955 2575 scope.go:117] "RemoveContainer" containerID="01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9" Jul 29 19:41:28.827158 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:28.827120 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:28.827258 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:28.827202 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:28.827258 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:28.827237 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:41:29.729372 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:29.729325 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:29.833041 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:29.833001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" event={"ID":"e3b61dad-7cbe-4ebd-b497-a17160dd779c","Type":"ContainerStarted","Data":"4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02"} Jul 29 19:41:30.344053 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:30.344011 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:30.345279 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:30.345246 2575 patch_prober.go:28] interesting pod/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 19:41:30.345444 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:30.345314 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 19:41:31.343909 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:31.343863 2575 patch_prober.go:28] interesting pod/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 19:41:31.344378 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:31.343969 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 19:41:32.344245 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:32.343822 2575 patch_prober.go:28] interesting pod/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 19:41:32.344245 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:32.343880 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 19:41:33.344273 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:33.344235 2575 patch_prober.go:28] interesting pod/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 29 19:41:33.344857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:33.344825 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 29 19:41:34.348024 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:34.347996 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:34.348397 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:34.348343 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:41:34.349065 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:41:34.349045 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:43:38.989268 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:38.989236 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c"] Jul 29 19:43:38.989763 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:38.989449 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" containerID="cri-o://060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed" gracePeriod=30 Jul 29 19:43:44.256117 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.256056 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:43:44.387389 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387350 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjq6k\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-kube-api-access-gjq6k\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.387389 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387395 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-certs\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.387660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387413 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-data\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.387660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387431 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-token\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.387660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387453 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-socket\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.387660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387585 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-podinfo\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.387660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387619 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-credential-socket\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.387911 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-envoy\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.387911 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387747 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee7e1a9-4076-42b5-baa5-61ce51014813-istiod-ca-cert\") pod \"6ee7e1a9-4076-42b5-baa5-61ce51014813\" (UID: \"6ee7e1a9-4076-42b5-baa5-61ce51014813\") " Jul 29 19:43:44.388103 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387739 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:43:44.388103 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.387882 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:43:44.388280 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.388040 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:43:44.388280 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.388156 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee7e1a9-4076-42b5-baa5-61ce51014813-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:43:44.388458 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.388433 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:43:44.389869 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.389847 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:43:44.389990 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.389973 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:43:44.390194 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.390173 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:43:44.390321 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.390302 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-kube-api-access-gjq6k" (OuterVolumeSpecName: "kube-api-access-gjq6k") pod "6ee7e1a9-4076-42b5-baa5-61ce51014813" (UID: "6ee7e1a9-4076-42b5-baa5-61ce51014813"). InnerVolumeSpecName "kube-api-access-gjq6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:43:44.489094 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489062 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.489094 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489087 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee7e1a9-4076-42b5-baa5-61ce51014813-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.489368 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489097 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjq6k\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-kube-api-access-gjq6k\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.489368 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489119 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.489368 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489130 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.489368 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489138 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.489368 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489146 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.489368 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489154 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee7e1a9-4076-42b5-baa5-61ce51014813-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.489368 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.489163 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee7e1a9-4076-42b5-baa5-61ce51014813-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:43:44.637543 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.637494 2575 generic.go:356] "Generic (PLEG): container finished" podID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerID="060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed" exitCode=0 Jul 29 19:43:44.637750 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.637567 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" Jul 29 19:43:44.637750 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.637586 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" event={"ID":"6ee7e1a9-4076-42b5-baa5-61ce51014813","Type":"ContainerDied","Data":"060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed"} Jul 29 19:43:44.637750 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.637654 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c" event={"ID":"6ee7e1a9-4076-42b5-baa5-61ce51014813","Type":"ContainerDied","Data":"49a67029d7f5fd053e66687ecb822b4b21209f89d82143fbe8019333d8e81d24"} Jul 29 19:43:44.637750 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.637685 2575 scope.go:117] "RemoveContainer" containerID="060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed" Jul 29 19:43:44.651593 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.651576 2575 scope.go:117] "RemoveContainer" containerID="1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf" Jul 29 19:43:44.660094 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.660075 2575 scope.go:117] "RemoveContainer" containerID="060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed" Jul 29 19:43:44.660382 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:43:44.660358 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed\": container with ID starting with 060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed not found: ID does not exist" containerID="060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed" Jul 29 19:43:44.660452 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.660396 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed"} err="failed to get container status \"060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed\": rpc error: code = NotFound desc = could not find container \"060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed\": container with ID starting with 060c806862ec6bfbb741ac967b367a0ce6550276e13bc8126eba62cfcf79f4ed not found: ID does not exist" Jul 29 19:43:44.660452 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.660421 2575 scope.go:117] "RemoveContainer" containerID="1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf" Jul 29 19:43:44.660689 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:43:44.660669 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf\": container with ID starting with 1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf not found: ID does not exist" containerID="1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf" Jul 29 19:43:44.660756 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.660693 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf"} err="failed to get container status \"1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf\": rpc error: code = NotFound desc = could not find container \"1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf\": container with ID starting with 1b6871a65fcd0207382903f41f3834ef5e630d00d84a6fc9fb93d5629e0644cf not found: ID does not exist" Jul 29 19:43:44.661956 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.661937 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c"] Jul 29 19:43:44.666014 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:44.665994 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-909676-openshift-default-568c49c9f6-fb47c"] Jul 29 19:43:45.550352 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:45.550306 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" path="/var/lib/kubelet/pods/6ee7e1a9-4076-42b5-baa5-61ce51014813/volumes" Jul 29 19:43:56.432083 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:56.432052 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt"] Jul 29 19:43:56.432595 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:56.432247 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" containerID="cri-o://4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02" gracePeriod=30 Jul 29 19:43:57.282778 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.282745 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc"] Jul 29 19:43:57.283195 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.283179 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" Jul 29 19:43:57.283239 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.283197 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" Jul 29 19:43:57.283239 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.283214 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" Jul 29 19:43:57.283239 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.283220 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" Jul 29 19:43:57.283329 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.283278 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" Jul 29 19:43:57.286301 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.286284 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.289169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.289147 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9e7a39-openshift-default-dockercfg-nvdzc\"" Jul 29 19:43:57.297657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.297634 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc"] Jul 29 19:43:57.414768 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-podinfo\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.414948 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414779 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-envoy\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.414948 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a088b4de-6764-4e7a-850e-f811ae441bb2-istiod-ca-cert\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.414948 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-token\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.414948 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95qd\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-kube-api-access-v95qd\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.414948 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-credential-socket\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.414948 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-socket\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.415159 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-data\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.415159 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.414993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-certs\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515511 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-podinfo\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-envoy\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a088b4de-6764-4e7a-850e-f811ae441bb2-istiod-ca-cert\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-token\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95qd\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-kube-api-access-v95qd\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-credential-socket\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-socket\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-data\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.515977 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.515863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-certs\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.516376 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.516127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-socket\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.516376 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.516199 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-credential-socket\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.516470 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.516374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-data\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.516470 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.516394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-certs\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.516470 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.516430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a088b4de-6764-4e7a-850e-f811ae441bb2-istiod-ca-cert\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.517962 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.517942 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-envoy\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.518111 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.518092 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-podinfo\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.523660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.523640 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-token\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.523738 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.523725 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95qd\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-kube-api-access-v95qd\") pod \"e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.597089 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.597009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:57.744655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.744629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc"] Jul 29 19:43:57.746875 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:43:57.746841 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda088b4de_6764_4e7a_850e_f811ae441bb2.slice/crio-02fd777a4f0f3bda017d8d9d1b9bf6707dd9062244506721ed10eee882d67768 WatchSource:0}: Error finding container 02fd777a4f0f3bda017d8d9d1b9bf6707dd9062244506721ed10eee882d67768: Status 404 returned error can't find the container with id 02fd777a4f0f3bda017d8d9d1b9bf6707dd9062244506721ed10eee882d67768 Jul 29 19:43:57.749394 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.749364 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:43:57.749469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.749434 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:43:57.749469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:57.749462 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:43:58.721169 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:58.721131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" event={"ID":"a088b4de-6764-4e7a-850e-f811ae441bb2","Type":"ContainerStarted","Data":"16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5"} Jul 29 19:43:58.721635 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:58.721192 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" event={"ID":"a088b4de-6764-4e7a-850e-f811ae441bb2","Type":"ContainerStarted","Data":"02fd777a4f0f3bda017d8d9d1b9bf6707dd9062244506721ed10eee882d67768"} Jul 29 19:43:58.747314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:58.747261 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" podStartSLOduration=1.747244238 podStartE2EDuration="1.747244238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:43:58.74658511 +0000 UTC m=+2619.763606203" watchObservedRunningTime="2026-07-29 19:43:58.747244238 +0000 UTC m=+2619.764265385" Jul 29 19:43:59.598034 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:59.598000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:59.602816 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:59.602795 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:59.727109 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:59.727069 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:43:59.727975 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:43:59.727954 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:44:01.689327 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.689304 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:44:01.741000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.740903 2575 generic.go:356] "Generic (PLEG): container finished" podID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerID="4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02" exitCode=0 Jul 29 19:44:01.741000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.740960 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" Jul 29 19:44:01.741000 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.740987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" event={"ID":"e3b61dad-7cbe-4ebd-b497-a17160dd779c","Type":"ContainerDied","Data":"4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02"} Jul 29 19:44:01.741292 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.741055 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt" event={"ID":"e3b61dad-7cbe-4ebd-b497-a17160dd779c","Type":"ContainerDied","Data":"e5005a49ea75dc8dc2284695d345000727b6666c411ccd8e956a87f11abc14ed"} Jul 29 19:44:01.741292 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.741098 2575 scope.go:117] "RemoveContainer" containerID="4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02" Jul 29 19:44:01.752628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.752607 2575 scope.go:117] "RemoveContainer" containerID="01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9" Jul 29 19:44:01.764017 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.763995 2575 scope.go:117] "RemoveContainer" containerID="4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02" Jul 29 19:44:01.764308 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:44:01.764285 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02\": container with ID starting with 4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02 not found: ID does not exist" containerID="4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02" Jul 29 19:44:01.764359 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.764317 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02"} err="failed to get container status \"4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02\": rpc error: code = NotFound desc = could not find container \"4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02\": container with ID starting with 4920faba7cec092d745c759056f8a12959f0942e1d635416573dc6b8fd64fc02 not found: ID does not exist" Jul 29 19:44:01.764359 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.764348 2575 scope.go:117] "RemoveContainer" containerID="01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9" Jul 29 19:44:01.764618 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:44:01.764599 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9\": container with ID starting with 01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9 not found: ID does not exist" containerID="01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9" Jul 29 19:44:01.764693 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.764624 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9"} err="failed to get container status \"01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9\": rpc error: code = NotFound desc = could not find container \"01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9\": container with ID starting with 01fb4b7ca2d995817452b8467fa59786c6efbb404bd2ea047f56f9add342bfd9 not found: ID does not exist" Jul 29 19:44:01.859736 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.859708 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92v9v\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-kube-api-access-92v9v\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.859885 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.859774 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-credential-socket\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.859885 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.859794 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-socket\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.859976 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.859899 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-podinfo\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.859976 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.859926 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-certs\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.859976 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.859960 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-data\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.860125 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860028 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-token\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.860125 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860067 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:01.860125 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860091 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:01.860125 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860113 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istiod-ca-cert\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.860317 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860183 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-envoy\") pod \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\" (UID: \"e3b61dad-7cbe-4ebd-b497-a17160dd779c\") " Jul 29 19:44:01.860317 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860236 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:01.860439 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860417 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:44:01.860676 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860653 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:01.860785 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860680 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:01.860785 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860695 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:01.860785 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860709 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:01.860953 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.860844 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:01.862233 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.862199 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:44:01.862350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.862308 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-kube-api-access-92v9v" (OuterVolumeSpecName: "kube-api-access-92v9v") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "kube-api-access-92v9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:44:01.862456 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.862434 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:44:01.862494 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.862457 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3b61dad-7cbe-4ebd-b497-a17160dd779c" (UID: "e3b61dad-7cbe-4ebd-b497-a17160dd779c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:01.961976 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.961945 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:01.961976 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.961973 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:01.961976 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.961982 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:01.962577 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.961989 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b61dad-7cbe-4ebd-b497-a17160dd779c-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:01.962577 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:01.961998 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92v9v\" (UniqueName: \"kubernetes.io/projected/e3b61dad-7cbe-4ebd-b497-a17160dd779c-kube-api-access-92v9v\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:02.071370 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:02.071338 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt"] Jul 29 19:44:02.077575 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:02.077544 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bf1c-openshift-default-557f898894-hvwpt"] Jul 29 19:44:03.550112 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:03.550077 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" path="/var/lib/kubelet/pods/e3b61dad-7cbe-4ebd-b497-a17160dd779c/volumes" Jul 29 19:44:11.103871 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.103808 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g"] Jul 29 19:44:11.104434 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.104405 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" Jul 29 19:44:11.104434 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.104428 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" Jul 29 19:44:11.104606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.104449 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" Jul 29 19:44:11.104606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.104456 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" Jul 29 19:44:11.104606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.104564 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" Jul 29 19:44:11.104606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.104574 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b61dad-7cbe-4ebd-b497-a17160dd779c" containerName="istio-proxy" Jul 29 19:44:11.104606 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.104582 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ee7e1a9-4076-42b5-baa5-61ce51014813" containerName="istio-proxy" Jul 29 19:44:11.109153 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.109138 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.112199 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.112183 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f9937e-openshift-default-dockercfg-2wckk\"" Jul 29 19:44:11.125300 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.125271 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g"] Jul 29 19:44:11.160916 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.160882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-envoy\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.161182 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.161159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-istio-token\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.161363 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.161345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5gt\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-kube-api-access-jp5gt\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.161527 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.161495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-socket\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.161719 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.161700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616cc4ed-f549-423f-9b56-986db97a22ea-istio-podinfo\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.161879 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.161863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616cc4ed-f549-423f-9b56-986db97a22ea-istiod-ca-cert\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.161997 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.161979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-data\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.162112 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.162096 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-credential-socket\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.162264 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.162249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-certs\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264132 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-certs\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264330 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-envoy\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264330 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-istio-token\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264330 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5gt\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-kube-api-access-jp5gt\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264330 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-socket\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264330 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616cc4ed-f549-423f-9b56-986db97a22ea-istio-podinfo\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264645 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616cc4ed-f549-423f-9b56-986db97a22ea-istiod-ca-cert\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264645 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-data\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.264645 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-credential-socket\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.265223 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.264879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-credential-socket\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.265223 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.265179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-socket\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.265450 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.265393 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-certs\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.265903 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.265864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-data\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.266118 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.266039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616cc4ed-f549-423f-9b56-986db97a22ea-istiod-ca-cert\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.267623 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.267583 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-envoy\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.267859 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.267838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616cc4ed-f549-423f-9b56-986db97a22ea-istio-podinfo\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.276906 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.276876 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5gt\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-kube-api-access-jp5gt\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.278946 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.278922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-istio-token\") pod \"e2e-api-b-f9937e-openshift-default-b87768c56-g872g\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.419434 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.419399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:11.611055 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.611023 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g"] Jul 29 19:44:11.612403 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:44:11.612378 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616cc4ed_f549_423f_9b56_986db97a22ea.slice/crio-51ba339a4b3e2aa414ec6821984b43f494d124c4045be1eddf0a7d0adb98c85e WatchSource:0}: Error finding container 51ba339a4b3e2aa414ec6821984b43f494d124c4045be1eddf0a7d0adb98c85e: Status 404 returned error can't find the container with id 51ba339a4b3e2aa414ec6821984b43f494d124c4045be1eddf0a7d0adb98c85e Jul 29 19:44:11.615095 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.615060 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:11.615179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.615131 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:11.615179 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.615162 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:11.805032 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.804895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" event={"ID":"616cc4ed-f549-423f-9b56-986db97a22ea","Type":"ContainerStarted","Data":"987cbd721768412e9279627362bcd9ca73015353fe1519a588ce54fb8625d56f"} Jul 29 19:44:11.805032 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.804952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" event={"ID":"616cc4ed-f549-423f-9b56-986db97a22ea","Type":"ContainerStarted","Data":"51ba339a4b3e2aa414ec6821984b43f494d124c4045be1eddf0a7d0adb98c85e"} Jul 29 19:44:11.832194 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:11.832131 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" podStartSLOduration=0.832110973 podStartE2EDuration="832.110973ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:44:11.827792447 +0000 UTC m=+2632.844813544" watchObservedRunningTime="2026-07-29 19:44:11.832110973 +0000 UTC m=+2632.849132068" Jul 29 19:44:12.420503 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:12.420465 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:12.426205 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:12.426177 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:12.812159 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:12.812074 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:12.813150 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:12.813117 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:14.728457 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:14.728374 2575 patch_prober.go:28] interesting pod/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 29 19:44:14.728914 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:14.728448 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 29 19:44:14.824941 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:14.824859 2575 generic.go:356] "Generic (PLEG): container finished" podID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerID="16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5" exitCode=0 Jul 29 19:44:14.825076 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:14.824940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" event={"ID":"a088b4de-6764-4e7a-850e-f811ae441bb2","Type":"ContainerDied","Data":"16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5"} Jul 29 19:44:14.825407 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:14.825389 2575 scope.go:117] "RemoveContainer" containerID="16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5" Jul 29 19:44:14.826563 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:14.826494 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:14.826678 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:14.826619 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:14.826678 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:14.826664 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:15.832918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:15.832887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" event={"ID":"a088b4de-6764-4e7a-850e-f811ae441bb2","Type":"ContainerStarted","Data":"4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7"} Jul 29 19:44:16.597360 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:16.597298 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:44:16.597600 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:16.597575 2575 patch_prober.go:28] interesting pod/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 29 19:44:16.597658 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:16.597627 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 29 19:44:17.598176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:17.598136 2575 patch_prober.go:28] interesting pod/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 29 19:44:17.598619 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:17.598195 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 29 19:44:18.598124 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:18.598088 2575 patch_prober.go:28] interesting pod/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 29 19:44:18.598307 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:18.598142 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 29 19:44:19.598221 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:19.598182 2575 patch_prober.go:28] interesting pod/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 29 19:44:19.598402 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:19.598248 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 29 19:44:20.602214 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:20.602174 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:44:20.602636 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:20.602615 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:44:20.603381 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:20.603364 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:44:28.854676 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:28.854638 2575 patch_prober.go:28] interesting pod/e2e-api-b-f9937e-openshift-default-b87768c56-g872g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 19:44:28.855065 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:28.854713 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 19:44:29.927830 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:29.927805 2575 generic.go:356] "Generic (PLEG): container finished" podID="616cc4ed-f549-423f-9b56-986db97a22ea" containerID="987cbd721768412e9279627362bcd9ca73015353fe1519a588ce54fb8625d56f" exitCode=0 Jul 29 19:44:29.928139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:29.927840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" event={"ID":"616cc4ed-f549-423f-9b56-986db97a22ea","Type":"ContainerDied","Data":"987cbd721768412e9279627362bcd9ca73015353fe1519a588ce54fb8625d56f"} Jul 29 19:44:29.928250 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:29.928237 2575 scope.go:117] "RemoveContainer" containerID="987cbd721768412e9279627362bcd9ca73015353fe1519a588ce54fb8625d56f" Jul 29 19:44:29.929349 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:29.929299 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:29.929459 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:29.929401 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:29.929459 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:29.929448 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:44:30.936937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:30.935874 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" event={"ID":"616cc4ed-f549-423f-9b56-986db97a22ea","Type":"ContainerStarted","Data":"76e6b9a543618bd76187576d58b9f006195940351978d1dd0521e658ec50aca5"} Jul 29 19:44:31.420602 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:31.420564 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:31.420833 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:31.420806 2575 patch_prober.go:28] interesting pod/e2e-api-b-f9937e-openshift-default-b87768c56-g872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 19:44:31.420920 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:31.420870 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 19:44:32.420223 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:32.420185 2575 patch_prober.go:28] interesting pod/e2e-api-b-f9937e-openshift-default-b87768c56-g872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 19:44:32.420632 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:32.420243 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 19:44:33.419846 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:33.419805 2575 patch_prober.go:28] interesting pod/e2e-api-b-f9937e-openshift-default-b87768c56-g872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 19:44:33.420083 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:33.419884 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 19:44:34.420038 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:34.419994 2575 patch_prober.go:28] interesting pod/e2e-api-b-f9937e-openshift-default-b87768c56-g872g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 29 19:44:34.420499 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:34.420068 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 29 19:44:35.423591 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:35.423561 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:35.423984 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:35.423962 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:35.424656 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:35.424637 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:44:49.241561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:49.241491 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc"] Jul 29 19:44:49.242030 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:49.241832 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" containerID="cri-o://4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7" gracePeriod=30 Jul 29 19:44:50.605439 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:50.605400 2575 patch_prober.go:28] interesting pod/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 19:44:50.605825 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:50.605458 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:44:54.502685 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.502627 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:44:54.625150 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625117 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-podinfo\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.625338 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625163 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-envoy\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.625338 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625178 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-certs\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.625338 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625216 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-token\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.625338 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625241 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v95qd\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-kube-api-access-v95qd\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.625338 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625289 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-data\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625340 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-socket\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625478 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-credential-socket\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625619 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a088b4de-6764-4e7a-850e-f811ae441bb2-istiod-ca-cert\") pod \"a088b4de-6764-4e7a-850e-f811ae441bb2\" (UID: \"a088b4de-6764-4e7a-850e-f811ae441bb2\") " Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625644 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625687 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.625829 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.626015 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.626038 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.626040 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a088b4de-6764-4e7a-850e-f811ae441bb2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:44:54.626167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.626051 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:54.626907 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.626883 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-data" (OuterVolumeSpecName: "istio-data") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:54.627415 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.627397 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:44:54.627664 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.627647 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-kube-api-access-v95qd" (OuterVolumeSpecName: "kube-api-access-v95qd") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "kube-api-access-v95qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:44:54.627869 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.627836 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:44:54.627869 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.627842 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a088b4de-6764-4e7a-850e-f811ae441bb2" (UID: "a088b4de-6764-4e7a-850e-f811ae441bb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:44:54.727099 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.727074 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:54.727099 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.727097 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:54.727265 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.727105 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:54.727265 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.727114 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v95qd\" (UniqueName: \"kubernetes.io/projected/a088b4de-6764-4e7a-850e-f811ae441bb2-kube-api-access-v95qd\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:54.727265 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.727123 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a088b4de-6764-4e7a-850e-f811ae441bb2-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:54.727265 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:54.727130 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a088b4de-6764-4e7a-850e-f811ae441bb2-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:44:55.088161 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.088130 2575 generic.go:356] "Generic (PLEG): container finished" podID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerID="4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7" exitCode=0 Jul 29 19:44:55.088337 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.088191 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" Jul 29 19:44:55.088337 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.088218 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" event={"ID":"a088b4de-6764-4e7a-850e-f811ae441bb2","Type":"ContainerDied","Data":"4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7"} Jul 29 19:44:55.088337 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.088274 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc" event={"ID":"a088b4de-6764-4e7a-850e-f811ae441bb2","Type":"ContainerDied","Data":"02fd777a4f0f3bda017d8d9d1b9bf6707dd9062244506721ed10eee882d67768"} Jul 29 19:44:55.088337 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.088297 2575 scope.go:117] "RemoveContainer" containerID="4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7" Jul 29 19:44:55.098740 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.098718 2575 scope.go:117] "RemoveContainer" containerID="16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5" Jul 29 19:44:55.109259 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.109240 2575 scope.go:117] "RemoveContainer" containerID="4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7" Jul 29 19:44:55.109608 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:44:55.109587 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7\": container with ID starting with 4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7 not found: ID does not exist" containerID="4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7" Jul 29 19:44:55.109712 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.109613 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7"} err="failed to get container status \"4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7\": rpc error: code = NotFound desc = could not find container \"4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7\": container with ID starting with 4a0c42ef1ababd570fb121b738511c0dbf01f5bd9ce49d5ee6f75e6edb4222f7 not found: ID does not exist" Jul 29 19:44:55.109712 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.109631 2575 scope.go:117] "RemoveContainer" containerID="16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5" Jul 29 19:44:55.109894 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:44:55.109868 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5\": container with ID starting with 16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5 not found: ID does not exist" containerID="16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5" Jul 29 19:44:55.110015 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.109898 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5"} err="failed to get container status \"16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5\": rpc error: code = NotFound desc = could not find container \"16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5\": container with ID starting with 16a613fc71bfb8c3c96d3e604568e92737ac4730cb5af9fea919d4d8a7c889f5 not found: ID does not exist" Jul 29 19:44:55.115104 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.115080 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc"] Jul 29 19:44:55.120565 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.120541 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9e7a39-openshift-default-546446dff6-t4rdc"] Jul 29 19:44:55.549936 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:44:55.549896 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" path="/var/lib/kubelet/pods/a088b4de-6764-4e7a-850e-f811ae441bb2/volumes" Jul 29 19:45:06.954554 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:06.954498 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g"] Jul 29 19:45:06.955002 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:06.954758 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" containerID="cri-o://76e6b9a543618bd76187576d58b9f006195940351978d1dd0521e658ec50aca5" gracePeriod=30 Jul 29 19:45:07.874607 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.874570 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7"] Jul 29 19:45:07.875011 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.874994 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" Jul 29 19:45:07.875011 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.875010 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" Jul 29 19:45:07.875176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.875017 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" Jul 29 19:45:07.875176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.875023 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" Jul 29 19:45:07.875176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.875105 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" Jul 29 19:45:07.875176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.875113 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a088b4de-6764-4e7a-850e-f811ae441bb2" containerName="istio-proxy" Jul 29 19:45:07.878471 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.878452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.881404 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.881385 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-8e241a-openshift-default-dockercfg-2d9h2\"" Jul 29 19:45:07.900188 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.900166 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7"] Jul 29 19:45:07.958742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.958693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-socket\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.959219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.958745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxxm\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-kube-api-access-ssxxm\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.959219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.958810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-podinfo\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.959219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.958850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-certs\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.959219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.958879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-token\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.959219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.958904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-envoy\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.959219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.958926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-credential-socket\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.959219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.958961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-data\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:07.959219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:07.959074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ea91b-87cc-435a-887e-a542be0409d5-istiod-ca-cert\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.059836 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.059806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-socket\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.059836 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.059839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxxm\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-kube-api-access-ssxxm\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.059871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-podinfo\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.059893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-certs\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.059916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-token\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.059941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-envoy\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.059963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-credential-socket\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.059994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-data\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.060085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ea91b-87cc-435a-887e-a542be0409d5-istiod-ca-cert\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.060309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-certs\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.060350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-socket\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.060434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-credential-socket\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060666 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.060477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-data\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.060809 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.060787 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ea91b-87cc-435a-887e-a542be0409d5-istiod-ca-cert\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.062715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.062689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-podinfo\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.062807 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.062781 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-envoy\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.068207 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.068184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-token\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.068287 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.068189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxxm\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-kube-api-access-ssxxm\") pod \"e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.189130 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.189046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:08.332458 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.332431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7"] Jul 29 19:45:08.333877 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:45:08.333849 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7ea91b_87cc_435a_887e_a542be0409d5.slice/crio-f64655559c3b4ad3a598b03ed952c7628951c3a7b1e5e5ab60b63b16959e18d4 WatchSource:0}: Error finding container f64655559c3b4ad3a598b03ed952c7628951c3a7b1e5e5ab60b63b16959e18d4: Status 404 returned error can't find the container with id f64655559c3b4ad3a598b03ed952c7628951c3a7b1e5e5ab60b63b16959e18d4 Jul 29 19:45:08.336156 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.336121 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:08.336265 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.336204 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:08.336265 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:08.336242 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:09.180099 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:09.180060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" event={"ID":"8e7ea91b-87cc-435a-887e-a542be0409d5","Type":"ContainerStarted","Data":"55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642"} Jul 29 19:45:09.180590 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:09.180158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" event={"ID":"8e7ea91b-87cc-435a-887e-a542be0409d5","Type":"ContainerStarted","Data":"f64655559c3b4ad3a598b03ed952c7628951c3a7b1e5e5ab60b63b16959e18d4"} Jul 29 19:45:09.189210 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:09.189190 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:09.190835 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:09.190810 2575 patch_prober.go:28] interesting pod/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 29 19:45:09.190942 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:09.190868 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 29 19:45:09.206343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:09.206295 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" podStartSLOduration=2.206280381 podStartE2EDuration="2.206280381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:09.203892171 +0000 UTC m=+2690.220913268" watchObservedRunningTime="2026-07-29 19:45:09.206280381 +0000 UTC m=+2690.223301477" Jul 29 19:45:10.193115 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:10.193086 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:11.192672 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:11.192642 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:11.193724 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:11.193702 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:12.201106 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.201079 2575 generic.go:356] "Generic (PLEG): container finished" podID="616cc4ed-f549-423f-9b56-986db97a22ea" containerID="76e6b9a543618bd76187576d58b9f006195940351978d1dd0521e658ec50aca5" exitCode=0 Jul 29 19:45:12.201510 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.201152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" event={"ID":"616cc4ed-f549-423f-9b56-986db97a22ea","Type":"ContainerDied","Data":"76e6b9a543618bd76187576d58b9f006195940351978d1dd0521e658ec50aca5"} Jul 29 19:45:12.201510 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.201220 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" event={"ID":"616cc4ed-f549-423f-9b56-986db97a22ea","Type":"ContainerDied","Data":"51ba339a4b3e2aa414ec6821984b43f494d124c4045be1eddf0a7d0adb98c85e"} Jul 29 19:45:12.201510 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.201246 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51ba339a4b3e2aa414ec6821984b43f494d124c4045be1eddf0a7d0adb98c85e" Jul 29 19:45:12.201510 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.201266 2575 scope.go:117] "RemoveContainer" containerID="987cbd721768412e9279627362bcd9ca73015353fe1519a588ce54fb8625d56f" Jul 29 19:45:12.214480 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:45:12.214455 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987cbd721768412e9279627362bcd9ca73015353fe1519a588ce54fb8625d56f\": container with ID starting with 987cbd721768412e9279627362bcd9ca73015353fe1519a588ce54fb8625d56f not found: ID does not exist" containerID="987cbd721768412e9279627362bcd9ca73015353fe1519a588ce54fb8625d56f" Jul 29 19:45:12.217992 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.217952 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:45:12.300402 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.300362 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-istio-token\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.300616 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.300423 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-credential-socket\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.300811 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.300462 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-socket\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.300937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.300778 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:45:12.300937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.300898 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-certs\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.300937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.300773 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:45:12.301127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.300944 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-data\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.301127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301011 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616cc4ed-f549-423f-9b56-986db97a22ea-istiod-ca-cert\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.301127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301039 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp5gt\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-kube-api-access-jp5gt\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.301127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301149 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-envoy\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.301447 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301186 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:45:12.301447 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301227 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616cc4ed-f549-423f-9b56-986db97a22ea-istio-podinfo\") pod \"616cc4ed-f549-423f-9b56-986db97a22ea\" (UID: \"616cc4ed-f549-423f-9b56-986db97a22ea\") " Jul 29 19:45:12.301583 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301542 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616cc4ed-f549-423f-9b56-986db97a22ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:45:12.302144 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301789 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:12.302144 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301812 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:12.302144 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301828 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:12.302144 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.301841 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616cc4ed-f549-423f-9b56-986db97a22ea-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:12.302404 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.302133 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:45:12.303589 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.303487 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:45:12.304666 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.304641 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-kube-api-access-jp5gt" (OuterVolumeSpecName: "kube-api-access-jp5gt") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "kube-api-access-jp5gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:45:12.304857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.304829 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/616cc4ed-f549-423f-9b56-986db97a22ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:45:12.304857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.304849 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "616cc4ed-f549-423f-9b56-986db97a22ea" (UID: "616cc4ed-f549-423f-9b56-986db97a22ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:45:12.403054 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.403016 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp5gt\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-kube-api-access-jp5gt\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:12.403054 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.403046 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:12.403054 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.403058 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616cc4ed-f549-423f-9b56-986db97a22ea-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:12.403323 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.403069 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616cc4ed-f549-423f-9b56-986db97a22ea-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:12.403323 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:12.403080 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616cc4ed-f549-423f-9b56-986db97a22ea-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:45:13.208327 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:13.208293 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g" Jul 29 19:45:13.233861 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:13.233834 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g"] Jul 29 19:45:13.238554 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:13.238510 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f9937e-openshift-default-b87768c56-g872g"] Jul 29 19:45:13.549120 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:13.549030 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" path="/var/lib/kubelet/pods/616cc4ed-f549-423f-9b56-986db97a22ea/volumes" Jul 29 19:45:24.284490 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:24.284456 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerID="55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642" exitCode=0 Jul 29 19:45:24.284910 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:24.284548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" event={"ID":"8e7ea91b-87cc-435a-887e-a542be0409d5","Type":"ContainerDied","Data":"55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642"} Jul 29 19:45:24.284952 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:24.284928 2575 scope.go:117] "RemoveContainer" containerID="55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642" Jul 29 19:45:24.286007 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:24.285979 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:24.286068 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:24.286047 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:24.286109 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:24.286077 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:25.291917 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:25.291882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" event={"ID":"8e7ea91b-87cc-435a-887e-a542be0409d5","Type":"ContainerStarted","Data":"6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14"} Jul 29 19:45:26.189971 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:26.189934 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:26.191143 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:26.191116 2575 patch_prober.go:28] interesting pod/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 29 19:45:26.191244 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:26.191165 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 29 19:45:26.193267 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:26.193248 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:27.190349 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:27.190312 2575 patch_prober.go:28] interesting pod/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 29 19:45:27.190804 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:27.190376 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 29 19:45:28.189849 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:28.189812 2575 patch_prober.go:28] interesting pod/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 29 19:45:28.190035 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:28.189866 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 29 19:45:29.195232 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:29.195205 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:29.196315 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:29.196290 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:45:36.699139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.699109 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl"] Jul 29 19:45:36.699597 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.699583 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" Jul 29 19:45:36.699597 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.699599 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" Jul 29 19:45:36.699697 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.699688 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" Jul 29 19:45:36.699736 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.699701 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" Jul 29 19:45:36.702701 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.702686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.705418 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.705399 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-808386-openshift-default-dockercfg-hpnnp\"" Jul 29 19:45:36.720719 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.720685 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl"] Jul 29 19:45:36.853104 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-certs\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.853104 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-credential-socket\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.853343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-data\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.853343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853155 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-token\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.853343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853240 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e52a964-68ef-4b53-b13d-bec6ae1832da-istiod-ca-cert\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.853343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwpn\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-kube-api-access-pgwpn\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.853343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-envoy\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.853343 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-socket\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.853667 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.853433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-podinfo\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954464 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-podinfo\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954464 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-certs\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954464 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-credential-socket\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-data\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-token\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e52a964-68ef-4b53-b13d-bec6ae1832da-istiod-ca-cert\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwpn\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-kube-api-access-pgwpn\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-envoy\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.954772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954693 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-socket\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.955127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954955 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-credential-socket\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.955127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.954993 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-certs\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.955127 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.955051 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-data\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.955279 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.955263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-socket\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.955632 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.955609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e52a964-68ef-4b53-b13d-bec6ae1832da-istiod-ca-cert\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.957007 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.956992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-podinfo\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.957118 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.957101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-envoy\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.963943 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.963918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-token\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:36.964057 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:36.964008 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwpn\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-kube-api-access-pgwpn\") pod \"e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:37.013104 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:37.013071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:37.156747 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:37.156717 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl"] Jul 29 19:45:37.158172 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:45:37.158147 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e52a964_68ef_4b53_b13d_bec6ae1832da.slice/crio-a105337dd315b1fcc9b446dab356391c970994583359ca13ebc0666c03645e16 WatchSource:0}: Error finding container a105337dd315b1fcc9b446dab356391c970994583359ca13ebc0666c03645e16: Status 404 returned error can't find the container with id a105337dd315b1fcc9b446dab356391c970994583359ca13ebc0666c03645e16 Jul 29 19:45:37.160163 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:37.160128 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:37.160252 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:37.160205 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:37.160252 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:37.160247 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:37.361647 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:37.361611 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" event={"ID":"6e52a964-68ef-4b53-b13d-bec6ae1832da","Type":"ContainerStarted","Data":"9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3"} Jul 29 19:45:37.361823 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:37.361671 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" event={"ID":"6e52a964-68ef-4b53-b13d-bec6ae1832da","Type":"ContainerStarted","Data":"a105337dd315b1fcc9b446dab356391c970994583359ca13ebc0666c03645e16"} Jul 29 19:45:37.383459 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:37.383407 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" podStartSLOduration=1.383392436 podStartE2EDuration="1.383392436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:37.381344984 +0000 UTC m=+2718.398366080" watchObservedRunningTime="2026-07-29 19:45:37.383392436 +0000 UTC m=+2718.400413534" Jul 29 19:45:38.013604 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:38.013550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:38.018082 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:38.018060 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:38.367598 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:38.367489 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:38.368626 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:38.368603 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:53.709208 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:53.709025 2575 patch_prober.go:28] interesting pod/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": read tcp 10.132.0.2:46972->10.132.0.71:15021: read: connection reset by peer" start-of-body= Jul 29 19:45:53.709208 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:53.709126 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": read tcp 10.132.0.2:46972->10.132.0.71:15021: read: connection reset by peer" Jul 29 19:45:54.464839 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:54.464809 2575 generic.go:356] "Generic (PLEG): container finished" podID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerID="9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3" exitCode=0 Jul 29 19:45:54.465011 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:54.464882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" event={"ID":"6e52a964-68ef-4b53-b13d-bec6ae1832da","Type":"ContainerDied","Data":"9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3"} Jul 29 19:45:54.465299 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:54.465285 2575 scope.go:117] "RemoveContainer" containerID="9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3" Jul 29 19:45:54.466237 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:54.466209 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:54.466305 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:54.466278 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:54.466344 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:54.466307 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:45:55.471993 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:55.471955 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" event={"ID":"6e52a964-68ef-4b53-b13d-bec6ae1832da","Type":"ContainerStarted","Data":"d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015"} Jul 29 19:45:56.013774 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:56.013738 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:45:56.014007 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:56.013984 2575 patch_prober.go:28] interesting pod/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 29 19:45:56.014053 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:56.014036 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 29 19:45:57.013895 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:57.013861 2575 patch_prober.go:28] interesting pod/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 29 19:45:57.014280 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:57.013914 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 29 19:45:58.013582 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:58.013545 2575 patch_prober.go:28] interesting pod/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 29 19:45:58.013758 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:58.013597 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 29 19:45:59.014588 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:59.014550 2575 patch_prober.go:28] interesting pod/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 29 19:45:59.014965 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:45:59.014606 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 29 19:46:00.017237 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:46:00.017206 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:46:00.017628 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:46:00.017455 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:46:00.018118 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:46:00.018101 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:50:51.398856 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.398826 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v"] Jul 29 19:50:51.399306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.399252 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" Jul 29 19:50:51.399306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.399265 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="616cc4ed-f549-423f-9b56-986db97a22ea" containerName="istio-proxy" Jul 29 19:50:51.402655 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.402638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.406428 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.406405 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e241a\"/\"default-dockercfg-s79p8\"" Jul 29 19:50:51.406554 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.406405 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e241a\"/\"sub-test-model-8e241a-kserve-self-signed-certs\"" Jul 29 19:50:51.406650 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.406630 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e241a\"/\"openshift-service-ca.crt\"" Jul 29 19:50:51.406716 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.406702 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e241a\"/\"kube-root-ca.crt\"" Jul 29 19:50:51.415687 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.415663 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v"] Jul 29 19:50:51.517003 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.516968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a8e549-86af-44ec-9872-235179e0e2d9-tls-certs\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.517003 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.517001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-home\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.517194 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.517021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-model-cache\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.517194 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.517112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-dshm\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.517194 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.517142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjg48\" (UniqueName: \"kubernetes.io/projected/59a8e549-86af-44ec-9872-235179e0e2d9-kube-api-access-vjg48\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.517303 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.517243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-tmp-dir\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-tmp-dir\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618643 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a8e549-86af-44ec-9872-235179e0e2d9-tls-certs\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618643 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-home\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618643 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-model-cache\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618643 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-dshm\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618643 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjg48\" (UniqueName: \"kubernetes.io/projected/59a8e549-86af-44ec-9872-235179e0e2d9-kube-api-access-vjg48\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618965 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-home\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618965 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618867 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-tmp-dir\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.618965 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.618910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-model-cache\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.620764 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.620745 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-dshm\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.620881 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.620864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a8e549-86af-44ec-9872-235179e0e2d9-tls-certs\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.630428 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.630408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjg48\" (UniqueName: \"kubernetes.io/projected/59a8e549-86af-44ec-9872-235179e0e2d9-kube-api-access-vjg48\") pod \"sub-test-model-8e241a-kserve-744bc677cc-w9g8v\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.712258 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.712184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:50:51.858812 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:51.858783 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v"] Jul 29 19:50:51.861087 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:50:51.861061 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59a8e549_86af_44ec_9872_235179e0e2d9.slice/crio-a3def22bb93944a3101c1da2a255a52a7b2f6f570e4a0af6453a2f87f205e638 WatchSource:0}: Error finding container a3def22bb93944a3101c1da2a255a52a7b2f6f570e4a0af6453a2f87f205e638: Status 404 returned error can't find the container with id a3def22bb93944a3101c1da2a255a52a7b2f6f570e4a0af6453a2f87f205e638 Jul 29 19:50:52.182219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:52.182184 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" event={"ID":"59a8e549-86af-44ec-9872-235179e0e2d9","Type":"ContainerStarted","Data":"10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042"} Jul 29 19:50:52.182384 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:52.182227 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" event={"ID":"59a8e549-86af-44ec-9872-235179e0e2d9","Type":"ContainerStarted","Data":"a3def22bb93944a3101c1da2a255a52a7b2f6f570e4a0af6453a2f87f205e638"} Jul 29 19:50:52.205973 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:50:52.205924 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" podStartSLOduration=1.205909571 podStartE2EDuration="1.205909571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:50:52.204148589 +0000 UTC m=+3033.221169684" watchObservedRunningTime="2026-07-29 19:50:52.205909571 +0000 UTC m=+3033.222930665" Jul 29 19:51:01.712467 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:51:01.712432 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:51:01.712954 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:51:01.712481 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:51:01.724858 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:51:01.724833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:51:02.251772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:51:02.251742 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:51:19.866545 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:51:19.866455 2575 scope.go:117] "RemoveContainer" containerID="76e6b9a543618bd76187576d58b9f006195940351978d1dd0521e658ec50aca5" Jul 29 19:52:21.484590 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:21.484556 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v"] Jul 29 19:52:36.504243 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.504201 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" podUID="59a8e549-86af-44ec-9872-235179e0e2d9" containerName="main" containerID="cri-o://10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042" gracePeriod=45 Jul 29 19:52:36.761175 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.761120 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:52:36.780589 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.780549 2575 generic.go:356] "Generic (PLEG): container finished" podID="59a8e549-86af-44ec-9872-235179e0e2d9" containerID="10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042" exitCode=0 Jul 29 19:52:36.780764 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.780600 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" Jul 29 19:52:36.780764 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.780616 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" event={"ID":"59a8e549-86af-44ec-9872-235179e0e2d9","Type":"ContainerDied","Data":"10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042"} Jul 29 19:52:36.780764 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.780651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v" event={"ID":"59a8e549-86af-44ec-9872-235179e0e2d9","Type":"ContainerDied","Data":"a3def22bb93944a3101c1da2a255a52a7b2f6f570e4a0af6453a2f87f205e638"} Jul 29 19:52:36.780764 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.780673 2575 scope.go:117] "RemoveContainer" containerID="10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042" Jul 29 19:52:36.790438 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.790417 2575 scope.go:117] "RemoveContainer" containerID="10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042" Jul 29 19:52:36.790757 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:52:36.790738 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042\": container with ID starting with 10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042 not found: ID does not exist" containerID="10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042" Jul 29 19:52:36.790814 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.790767 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042"} err="failed to get container status \"10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042\": rpc error: code = NotFound desc = could not find container \"10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042\": container with ID starting with 10dcb9d3906a6ab94d9100cc5de7753467b8dca452cc4d5fa045041c65099042 not found: ID does not exist" Jul 29 19:52:36.841857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.841820 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-tmp-dir\") pod \"59a8e549-86af-44ec-9872-235179e0e2d9\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " Jul 29 19:52:36.841857 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.841864 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a8e549-86af-44ec-9872-235179e0e2d9-tls-certs\") pod \"59a8e549-86af-44ec-9872-235179e0e2d9\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " Jul 29 19:52:36.842101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.841908 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjg48\" (UniqueName: \"kubernetes.io/projected/59a8e549-86af-44ec-9872-235179e0e2d9-kube-api-access-vjg48\") pod \"59a8e549-86af-44ec-9872-235179e0e2d9\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " Jul 29 19:52:36.842101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.841976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-home\") pod \"59a8e549-86af-44ec-9872-235179e0e2d9\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " Jul 29 19:52:36.842101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.842002 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-model-cache\") pod \"59a8e549-86af-44ec-9872-235179e0e2d9\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " Jul 29 19:52:36.842101 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.842093 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-dshm\") pod \"59a8e549-86af-44ec-9872-235179e0e2d9\" (UID: \"59a8e549-86af-44ec-9872-235179e0e2d9\") " Jul 29 19:52:36.842306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.842146 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59a8e549-86af-44ec-9872-235179e0e2d9" (UID: "59a8e549-86af-44ec-9872-235179e0e2d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:52:36.842410 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.842380 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "59a8e549-86af-44ec-9872-235179e0e2d9" (UID: "59a8e549-86af-44ec-9872-235179e0e2d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:52:36.842567 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.842441 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:52:36.842567 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.842541 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-home" (OuterVolumeSpecName: "home") pod "59a8e549-86af-44ec-9872-235179e0e2d9" (UID: "59a8e549-86af-44ec-9872-235179e0e2d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:52:36.844306 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.844280 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a8e549-86af-44ec-9872-235179e0e2d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59a8e549-86af-44ec-9872-235179e0e2d9" (UID: "59a8e549-86af-44ec-9872-235179e0e2d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:52:36.844406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.844315 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a8e549-86af-44ec-9872-235179e0e2d9-kube-api-access-vjg48" (OuterVolumeSpecName: "kube-api-access-vjg48") pod "59a8e549-86af-44ec-9872-235179e0e2d9" (UID: "59a8e549-86af-44ec-9872-235179e0e2d9"). InnerVolumeSpecName "kube-api-access-vjg48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:52:36.844406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.844335 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-dshm" (OuterVolumeSpecName: "dshm") pod "59a8e549-86af-44ec-9872-235179e0e2d9" (UID: "59a8e549-86af-44ec-9872-235179e0e2d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:52:36.942978 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.942946 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:52:36.942978 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.942975 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a8e549-86af-44ec-9872-235179e0e2d9-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:52:36.942978 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.942987 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjg48\" (UniqueName: \"kubernetes.io/projected/59a8e549-86af-44ec-9872-235179e0e2d9-kube-api-access-vjg48\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:52:36.943207 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.942996 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:52:36.943207 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:36.943006 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a8e549-86af-44ec-9872-235179e0e2d9-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:52:37.103715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:37.103661 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v"] Jul 29 19:52:37.106147 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:37.106125 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8e241a/sub-test-model-8e241a-kserve-744bc677cc-w9g8v"] Jul 29 19:52:37.551385 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:52:37.551346 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a8e549-86af-44ec-9872-235179e0e2d9" path="/var/lib/kubelet/pods/59a8e549-86af-44ec-9872-235179e0e2d9/volumes" Jul 29 19:54:09.218003 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.217970 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7638-140520098586160"] Jul 29 19:54:09.218460 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.218393 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a8e549-86af-44ec-9872-235179e0e2d9" containerName="main" Jul 29 19:54:09.218460 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.218406 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a8e549-86af-44ec-9872-235179e0e2d9" containerName="main" Jul 29 19:54:09.218560 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.218488 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a8e549-86af-44ec-9872-235179e0e2d9" containerName="main" Jul 29 19:54:09.221742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.221725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098586160" Jul 29 19:54:09.227535 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.224905 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhphn\"" Jul 29 19:54:09.229915 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.229884 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-140520098586160"] Jul 29 19:54:09.360601 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.360558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwrw\" (UniqueName: \"kubernetes.io/projected/1613294e-2849-45bd-aeec-ae25383208c3-kube-api-access-wbwrw\") pod \"test-curl-7638-140520098586160\" (UID: \"1613294e-2849-45bd-aeec-ae25383208c3\") " pod="opendatahub/test-curl-7638-140520098586160" Jul 29 19:54:09.462202 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.462164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwrw\" (UniqueName: \"kubernetes.io/projected/1613294e-2849-45bd-aeec-ae25383208c3-kube-api-access-wbwrw\") pod \"test-curl-7638-140520098586160\" (UID: \"1613294e-2849-45bd-aeec-ae25383208c3\") " pod="opendatahub/test-curl-7638-140520098586160" Jul 29 19:54:09.470969 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.470914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwrw\" (UniqueName: \"kubernetes.io/projected/1613294e-2849-45bd-aeec-ae25383208c3-kube-api-access-wbwrw\") pod \"test-curl-7638-140520098586160\" (UID: \"1613294e-2849-45bd-aeec-ae25383208c3\") " pod="opendatahub/test-curl-7638-140520098586160" Jul 29 19:54:09.538216 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.538175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098586160" Jul 29 19:54:09.677491 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.677464 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-140520098586160"] Jul 29 19:54:09.679197 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:54:09.679172 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1613294e_2849_45bd_aeec_ae25383208c3.slice/crio-5f0584d2547bb899c8c828b9730726c2d13f6687676de0bbb0cbb1832c6192ff WatchSource:0}: Error finding container 5f0584d2547bb899c8c828b9730726c2d13f6687676de0bbb0cbb1832c6192ff: Status 404 returned error can't find the container with id 5f0584d2547bb899c8c828b9730726c2d13f6687676de0bbb0cbb1832c6192ff Jul 29 19:54:09.680584 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:09.680544 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:54:10.342324 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:10.342276 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-140520098586160" event={"ID":"1613294e-2849-45bd-aeec-ae25383208c3","Type":"ContainerStarted","Data":"5f0584d2547bb899c8c828b9730726c2d13f6687676de0bbb0cbb1832c6192ff"} Jul 29 19:54:11.349822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:11.349787 2575 generic.go:356] "Generic (PLEG): container finished" podID="1613294e-2849-45bd-aeec-ae25383208c3" containerID="5e695fdaee085d4f2c6be64cfe9803dd3cb5893c9cf443f198c06997b8f2d706" exitCode=0 Jul 29 19:54:11.350191 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:11.349875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-140520098586160" event={"ID":"1613294e-2849-45bd-aeec-ae25383208c3","Type":"ContainerDied","Data":"5e695fdaee085d4f2c6be64cfe9803dd3cb5893c9cf443f198c06997b8f2d706"} Jul 29 19:54:12.492432 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:12.492408 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098586160" Jul 29 19:54:12.528323 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:12.528295 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7638-140520098586160_1613294e-2849-45bd-aeec-ae25383208c3/test-curl-7638-140520098586160/0.log" Jul 29 19:54:12.552356 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:12.552323 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7638-140520098586160"] Jul 29 19:54:12.556201 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:12.556178 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7638-140520098586160"] Jul 29 19:54:12.592809 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:12.592776 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbwrw\" (UniqueName: \"kubernetes.io/projected/1613294e-2849-45bd-aeec-ae25383208c3-kube-api-access-wbwrw\") pod \"1613294e-2849-45bd-aeec-ae25383208c3\" (UID: \"1613294e-2849-45bd-aeec-ae25383208c3\") " Jul 29 19:54:12.595042 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:12.595016 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1613294e-2849-45bd-aeec-ae25383208c3-kube-api-access-wbwrw" (OuterVolumeSpecName: "kube-api-access-wbwrw") pod "1613294e-2849-45bd-aeec-ae25383208c3" (UID: "1613294e-2849-45bd-aeec-ae25383208c3"). InnerVolumeSpecName "kube-api-access-wbwrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:54:12.694443 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:12.694413 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbwrw\" (UniqueName: \"kubernetes.io/projected/1613294e-2849-45bd-aeec-ae25383208c3-kube-api-access-wbwrw\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:54:13.362854 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:13.362824 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098586160" Jul 29 19:54:13.363037 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:13.362867 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f0584d2547bb899c8c828b9730726c2d13f6687676de0bbb0cbb1832c6192ff" Jul 29 19:54:13.373090 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:54:13.373062 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-140520098586160\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="1613294e-2849-45bd-aeec-ae25383208c3" pod="opendatahub/test-curl-7638-140520098586160" Jul 29 19:54:13.550003 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:13.549970 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1613294e-2849-45bd-aeec-ae25383208c3" path="/var/lib/kubelet/pods/1613294e-2849-45bd-aeec-ae25383208c3/volumes" Jul 29 19:54:21.947707 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:21.947674 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7638-140520098651136"] Jul 29 19:54:21.948103 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:21.948090 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1613294e-2849-45bd-aeec-ae25383208c3" containerName="test-curl-7638-140520098586160" Jul 29 19:54:21.948103 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:21.948104 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1613294e-2849-45bd-aeec-ae25383208c3" containerName="test-curl-7638-140520098586160" Jul 29 19:54:21.948195 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:21.948185 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1613294e-2849-45bd-aeec-ae25383208c3" containerName="test-curl-7638-140520098586160" Jul 29 19:54:21.951542 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:21.951507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098651136" Jul 29 19:54:21.954143 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:21.954124 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhphn\"" Jul 29 19:54:21.958060 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:21.958034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-140520098651136"] Jul 29 19:54:21.987788 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:21.987765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t62wm\" (UniqueName: \"kubernetes.io/projected/deac3192-03e2-4024-ba00-6ec589702f7a-kube-api-access-t62wm\") pod \"test-curl-7638-140520098651136\" (UID: \"deac3192-03e2-4024-ba00-6ec589702f7a\") " pod="opendatahub/test-curl-7638-140520098651136" Jul 29 19:54:22.089053 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:22.089022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t62wm\" (UniqueName: \"kubernetes.io/projected/deac3192-03e2-4024-ba00-6ec589702f7a-kube-api-access-t62wm\") pod \"test-curl-7638-140520098651136\" (UID: \"deac3192-03e2-4024-ba00-6ec589702f7a\") " pod="opendatahub/test-curl-7638-140520098651136" Jul 29 19:54:22.097481 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:22.097451 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t62wm\" (UniqueName: \"kubernetes.io/projected/deac3192-03e2-4024-ba00-6ec589702f7a-kube-api-access-t62wm\") pod \"test-curl-7638-140520098651136\" (UID: \"deac3192-03e2-4024-ba00-6ec589702f7a\") " pod="opendatahub/test-curl-7638-140520098651136" Jul 29 19:54:22.262745 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:22.262662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098651136" Jul 29 19:54:22.402808 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:22.402785 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-140520098651136"] Jul 29 19:54:22.404560 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:54:22.404532 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeac3192_03e2_4024_ba00_6ec589702f7a.slice/crio-5d56abb67c00baa00f1ed7d90435f7b9396d63af771968488a6b2cc4dc319b9b WatchSource:0}: Error finding container 5d56abb67c00baa00f1ed7d90435f7b9396d63af771968488a6b2cc4dc319b9b: Status 404 returned error can't find the container with id 5d56abb67c00baa00f1ed7d90435f7b9396d63af771968488a6b2cc4dc319b9b Jul 29 19:54:22.423914 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:22.423892 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-140520098651136" event={"ID":"deac3192-03e2-4024-ba00-6ec589702f7a","Type":"ContainerStarted","Data":"5d56abb67c00baa00f1ed7d90435f7b9396d63af771968488a6b2cc4dc319b9b"} Jul 29 19:54:23.431131 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:23.431097 2575 generic.go:356] "Generic (PLEG): container finished" podID="deac3192-03e2-4024-ba00-6ec589702f7a" containerID="e5efb77fe29c401f39554272e953016f8aa790efc1ae10725d85ae8132552286" exitCode=0 Jul 29 19:54:23.431585 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:23.431155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-140520098651136" event={"ID":"deac3192-03e2-4024-ba00-6ec589702f7a","Type":"ContainerDied","Data":"e5efb77fe29c401f39554272e953016f8aa790efc1ae10725d85ae8132552286"} Jul 29 19:54:24.574960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:24.574934 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098651136" Jul 29 19:54:24.595626 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:24.595529 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7638-140520098651136_deac3192-03e2-4024-ba00-6ec589702f7a/test-curl-7638-140520098651136/0.log" Jul 29 19:54:24.613877 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:24.613847 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t62wm\" (UniqueName: \"kubernetes.io/projected/deac3192-03e2-4024-ba00-6ec589702f7a-kube-api-access-t62wm\") pod \"deac3192-03e2-4024-ba00-6ec589702f7a\" (UID: \"deac3192-03e2-4024-ba00-6ec589702f7a\") " Jul 29 19:54:24.615910 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:24.615885 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deac3192-03e2-4024-ba00-6ec589702f7a-kube-api-access-t62wm" (OuterVolumeSpecName: "kube-api-access-t62wm") pod "deac3192-03e2-4024-ba00-6ec589702f7a" (UID: "deac3192-03e2-4024-ba00-6ec589702f7a"). InnerVolumeSpecName "kube-api-access-t62wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:54:24.621469 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:24.621440 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7638-140520098651136"] Jul 29 19:54:24.624199 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:24.624179 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7638-140520098651136"] Jul 29 19:54:24.714902 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:24.714835 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t62wm\" (UniqueName: \"kubernetes.io/projected/deac3192-03e2-4024-ba00-6ec589702f7a-kube-api-access-t62wm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:54:25.444486 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:25.444454 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-140520098651136" Jul 29 19:54:25.444486 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:25.444492 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d56abb67c00baa00f1ed7d90435f7b9396d63af771968488a6b2cc4dc319b9b" Jul 29 19:54:25.454826 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:54:25.454797 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-140520098651136\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="deac3192-03e2-4024-ba00-6ec589702f7a" pod="opendatahub/test-curl-7638-140520098651136" Jul 29 19:54:25.549460 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:25.549420 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deac3192-03e2-4024-ba00-6ec589702f7a" path="/var/lib/kubelet/pods/deac3192-03e2-4024-ba00-6ec589702f7a/volumes" Jul 29 19:54:59.586206 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.586173 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld"] Jul 29 19:54:59.586634 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.586625 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deac3192-03e2-4024-ba00-6ec589702f7a" containerName="test-curl-7638-140520098651136" Jul 29 19:54:59.586682 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.586638 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="deac3192-03e2-4024-ba00-6ec589702f7a" containerName="test-curl-7638-140520098651136" Jul 29 19:54:59.586718 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.586703 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="deac3192-03e2-4024-ba00-6ec589702f7a" containerName="test-curl-7638-140520098651136" Jul 29 19:54:59.591441 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.591424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.594211 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.594193 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f36365-openshift-default-dockercfg-ql2ps\"" Jul 29 19:54:59.600944 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.600923 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld"] Jul 29 19:54:59.660668 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.660631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-socket\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.660850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.660677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgvb\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-kube-api-access-zsgvb\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.660850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.660704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-certs\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.660850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.660753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a77f29-d9ee-49e0-8616-59e6b902227d-istiod-ca-cert\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.660850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.660817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-data\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.661089 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.660919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-podinfo\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.661089 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.660988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-credential-socket\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.661089 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.661018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-envoy\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.661089 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.661042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-token\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.762547 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.762493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgvb\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-kube-api-access-zsgvb\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.762715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.762559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-certs\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.762715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.762617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a77f29-d9ee-49e0-8616-59e6b902227d-istiod-ca-cert\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.762715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.762684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-data\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.762927 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.762738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-podinfo\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.762927 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.762788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-credential-socket\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.762927 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.762815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-envoy\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.762927 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.762837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-token\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.763135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.763000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-certs\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.763135 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.763051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-socket\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.763284 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.763262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-credential-socket\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.763386 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.763367 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-socket\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.763456 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.763376 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a77f29-d9ee-49e0-8616-59e6b902227d-istiod-ca-cert\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.763456 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.763420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-data\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.765007 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.764988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-envoy\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.765532 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.765500 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-podinfo\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.783107 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.783078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-token\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.783205 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.783189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgvb\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-kube-api-access-zsgvb\") pod \"e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:54:59.902385 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:54:59.902348 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:55:00.049474 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:00.049447 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld"] Jul 29 19:55:00.051714 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:55:00.051687 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a77f29_d9ee_49e0_8616_59e6b902227d.slice/crio-77b358a01ae7807b0bd255bef3e683bd99d59b6085327967ebc9dd8e4b54e700 WatchSource:0}: Error finding container 77b358a01ae7807b0bd255bef3e683bd99d59b6085327967ebc9dd8e4b54e700: Status 404 returned error can't find the container with id 77b358a01ae7807b0bd255bef3e683bd99d59b6085327967ebc9dd8e4b54e700 Jul 29 19:55:00.053901 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:00.053870 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:55:00.053948 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:00.053934 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:55:00.053993 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:00.053963 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:55:00.659715 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:00.659675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" event={"ID":"15a77f29-d9ee-49e0-8616-59e6b902227d","Type":"ContainerStarted","Data":"71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e"} Jul 29 19:55:00.660370 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:00.660341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" event={"ID":"15a77f29-d9ee-49e0-8616-59e6b902227d","Type":"ContainerStarted","Data":"77b358a01ae7807b0bd255bef3e683bd99d59b6085327967ebc9dd8e4b54e700"} Jul 29 19:55:00.691250 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:00.691182 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" podStartSLOduration=1.691168227 podStartE2EDuration="1.691168227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:55:00.689004853 +0000 UTC m=+3281.706025948" watchObservedRunningTime="2026-07-29 19:55:00.691168227 +0000 UTC m=+3281.708189323" Jul 29 19:55:00.903181 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:00.903153 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:55:01.907878 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:01.907845 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:55:02.670965 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:02.670936 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:55:02.671877 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:02.671853 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:55:20.792486 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:20.792453 2575 generic.go:356] "Generic (PLEG): container finished" podID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerID="71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e" exitCode=0 Jul 29 19:55:20.792916 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:20.792542 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" event={"ID":"15a77f29-d9ee-49e0-8616-59e6b902227d","Type":"ContainerDied","Data":"71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e"} Jul 29 19:55:20.793007 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:20.792991 2575 scope.go:117] "RemoveContainer" containerID="71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e" Jul 29 19:55:20.794111 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:20.794084 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:55:20.794170 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:20.794148 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:55:20.794207 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:20.794181 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 19:55:21.800557 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:21.800498 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" event={"ID":"15a77f29-d9ee-49e0-8616-59e6b902227d","Type":"ContainerStarted","Data":"7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f"} Jul 29 19:55:21.903391 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:21.903354 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:55:21.903609 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:21.903587 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 29 19:55:21.903660 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:21.903644 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 29 19:55:22.903131 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:22.903093 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 29 19:55:22.903537 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:22.903151 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 29 19:55:23.903776 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:23.903737 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 29 19:55:23.904153 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:23.903795 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 29 19:55:24.902960 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:24.902919 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 29 19:55:24.903124 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:24.902986 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 29 19:55:25.906855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:25.906820 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:55:25.907335 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:25.907187 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:55:25.907918 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:55:25.907898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:58:28.283770 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:28.283735 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld"] Jul 29 19:58:28.284261 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:28.283965 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" containerID="cri-o://7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f" gracePeriod=30 Jul 29 19:58:33.535022 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.534997 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:58:33.701279 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701241 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-data\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701459 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701330 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-credential-socket\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701459 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701402 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-socket\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701459 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701434 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-token\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701656 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701471 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-certs\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701656 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701497 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsgvb\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-kube-api-access-zsgvb\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701656 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701550 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-envoy\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701656 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701584 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-podinfo\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701683 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:33.701855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701776 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a77f29-d9ee-49e0-8616-59e6b902227d-istiod-ca-cert\") pod \"15a77f29-d9ee-49e0-8616-59e6b902227d\" (UID: \"15a77f29-d9ee-49e0-8616-59e6b902227d\") " Jul 29 19:58:33.701855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701783 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:33.701855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.701797 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:33.702104 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.702082 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a77f29-d9ee-49e0-8616-59e6b902227d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:58:33.702167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.702135 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.702167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.702150 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.702167 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.702162 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.702327 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.702185 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-data" (OuterVolumeSpecName: "istio-data") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:33.703814 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.703784 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-token" (OuterVolumeSpecName: "istio-token") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:58:33.703814 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.703790 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:58:33.703814 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.703798 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:33.703992 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.703871 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-kube-api-access-zsgvb" (OuterVolumeSpecName: "kube-api-access-zsgvb") pod "15a77f29-d9ee-49e0-8616-59e6b902227d" (UID: "15a77f29-d9ee-49e0-8616-59e6b902227d"). InnerVolumeSpecName "kube-api-access-zsgvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:58:33.803416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.803384 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.803416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.803411 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsgvb\" (UniqueName: \"kubernetes.io/projected/15a77f29-d9ee-49e0-8616-59e6b902227d-kube-api-access-zsgvb\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.803416 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.803420 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.803669 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.803428 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.803669 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.803438 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a77f29-d9ee-49e0-8616-59e6b902227d-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.803669 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.803445 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a77f29-d9ee-49e0-8616-59e6b902227d-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:33.941673 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.941637 2575 generic.go:356] "Generic (PLEG): container finished" podID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerID="7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f" exitCode=0 Jul 29 19:58:33.941855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.941699 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" Jul 29 19:58:33.941855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.941719 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" event={"ID":"15a77f29-d9ee-49e0-8616-59e6b902227d","Type":"ContainerDied","Data":"7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f"} Jul 29 19:58:33.941855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.941783 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld" event={"ID":"15a77f29-d9ee-49e0-8616-59e6b902227d","Type":"ContainerDied","Data":"77b358a01ae7807b0bd255bef3e683bd99d59b6085327967ebc9dd8e4b54e700"} Jul 29 19:58:33.941855 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.941818 2575 scope.go:117] "RemoveContainer" containerID="7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f" Jul 29 19:58:33.952014 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.951998 2575 scope.go:117] "RemoveContainer" containerID="71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e" Jul 29 19:58:33.960173 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.960152 2575 scope.go:117] "RemoveContainer" containerID="7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f" Jul 29 19:58:33.960439 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:58:33.960420 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f\": container with ID starting with 7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f not found: ID does not exist" containerID="7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f" Jul 29 19:58:33.960525 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.960447 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f"} err="failed to get container status \"7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f\": rpc error: code = NotFound desc = could not find container \"7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f\": container with ID starting with 7b20c31d13b1f7020fd7f9a908d1bfd7413faf73296a6ddaa7b4b42de764f66f not found: ID does not exist" Jul 29 19:58:33.960525 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.960466 2575 scope.go:117] "RemoveContainer" containerID="71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e" Jul 29 19:58:33.960703 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:58:33.960686 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e\": container with ID starting with 71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e not found: ID does not exist" containerID="71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e" Jul 29 19:58:33.960742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.960710 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e"} err="failed to get container status \"71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e\": rpc error: code = NotFound desc = could not find container \"71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e\": container with ID starting with 71f7d2ebd132f64453f14c650556ab0f97cd70764fcc2bed445ea1506f9aa77e not found: ID does not exist" Jul 29 19:58:33.965702 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.965680 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld"] Jul 29 19:58:33.969631 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:33.969610 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f36365-openshift-default-65cbc46dd6-9r2ld"] Jul 29 19:58:35.550139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:35.550104 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" path="/var/lib/kubelet/pods/15a77f29-d9ee-49e0-8616-59e6b902227d/volumes" Jul 29 19:58:45.682508 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:45.682409 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7"] Jul 29 19:58:45.682958 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:45.682677 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" containerID="cri-o://6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14" gracePeriod=30 Jul 29 19:58:50.935754 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.935725 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:58:50.958093 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssxxm\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-kube-api-access-ssxxm\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958238 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958089 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-socket\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958238 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958120 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-token\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958238 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958144 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-certs\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958238 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958186 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-credential-socket\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958462 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958266 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-data\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958462 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958327 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-podinfo\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958462 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958360 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-envoy\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958462 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958403 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ea91b-87cc-435a-887e-a542be0409d5-istiod-ca-cert\") pod \"8e7ea91b-87cc-435a-887e-a542be0409d5\" (UID: \"8e7ea91b-87cc-435a-887e-a542be0409d5\") " Jul 29 19:58:50.958691 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958628 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:50.958832 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958776 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:50.958832 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.958813 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:50.960292 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.959214 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:50.960292 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.959536 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7ea91b-87cc-435a-887e-a542be0409d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:58:50.960292 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.960030 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:50.961387 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.961357 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-kube-api-access-ssxxm" (OuterVolumeSpecName: "kube-api-access-ssxxm") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "kube-api-access-ssxxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:58:50.962548 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.962494 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:58:50.962772 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.962743 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:58:50.962919 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:50.962903 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e7ea91b-87cc-435a-887e-a542be0409d5" (UID: "8e7ea91b-87cc-435a-887e-a542be0409d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:58:51.059331 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.059296 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:51.059331 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.059327 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:51.059561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.059342 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:51.059561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.059356 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:51.059561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.059368 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:51.059561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.059376 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ea91b-87cc-435a-887e-a542be0409d5-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:51.059561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.059385 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ea91b-87cc-435a-887e-a542be0409d5-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:51.059561 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.059393 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxxm\" (UniqueName: \"kubernetes.io/projected/8e7ea91b-87cc-435a-887e-a542be0409d5-kube-api-access-ssxxm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:58:51.060439 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.060412 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerID="6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14" exitCode=0 Jul 29 19:58:51.060528 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.060448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" event={"ID":"8e7ea91b-87cc-435a-887e-a542be0409d5","Type":"ContainerDied","Data":"6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14"} Jul 29 19:58:51.060528 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.060468 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" Jul 29 19:58:51.060528 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.060503 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7" event={"ID":"8e7ea91b-87cc-435a-887e-a542be0409d5","Type":"ContainerDied","Data":"f64655559c3b4ad3a598b03ed952c7628951c3a7b1e5e5ab60b63b16959e18d4"} Jul 29 19:58:51.060639 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.060540 2575 scope.go:117] "RemoveContainer" containerID="6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14" Jul 29 19:58:51.070657 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.070638 2575 scope.go:117] "RemoveContainer" containerID="55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642" Jul 29 19:58:51.079191 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.079175 2575 scope.go:117] "RemoveContainer" containerID="6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14" Jul 29 19:58:51.079531 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:58:51.079491 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14\": container with ID starting with 6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14 not found: ID does not exist" containerID="6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14" Jul 29 19:58:51.079586 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.079537 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14"} err="failed to get container status \"6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14\": rpc error: code = NotFound desc = could not find container \"6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14\": container with ID starting with 6d399e36cf60a963de6bd47afc93cb10380fcffed6136207dc12abacec27bb14 not found: ID does not exist" Jul 29 19:58:51.079586 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.079563 2575 scope.go:117] "RemoveContainer" containerID="55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642" Jul 29 19:58:51.079967 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:58:51.079932 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642\": container with ID starting with 55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642 not found: ID does not exist" containerID="55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642" Jul 29 19:58:51.080062 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.079986 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642"} err="failed to get container status \"55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642\": rpc error: code = NotFound desc = could not find container \"55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642\": container with ID starting with 55488e883d103558f610f92fd8a81f56789e06e6be4d530ef529068445065642 not found: ID does not exist" Jul 29 19:58:51.082822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.082802 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7"] Jul 29 19:58:51.086603 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.086582 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-8e241a-openshift-default-c799dfb84-4cjs7"] Jul 29 19:58:51.551239 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:58:51.551209 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" path="/var/lib/kubelet/pods/8e7ea91b-87cc-435a-887e-a542be0409d5/volumes" Jul 29 19:59:03.187264 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:03.187230 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl"] Jul 29 19:59:03.187753 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:03.187478 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" containerID="cri-o://d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015" gracePeriod=30 Jul 29 19:59:03.855215 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:03.855181 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-565f6d5884-w59bh_e36391c1-e0a0-441a-9392-9b73db31f7ab/authorino/0.log" Jul 29 19:59:08.441983 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.441956 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:59:08.522223 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522150 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgwpn\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-kube-api-access-pgwpn\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522223 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522211 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-socket\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522396 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522271 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-data\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522396 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522306 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-token\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522396 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522329 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e52a964-68ef-4b53-b13d-bec6ae1832da-istiod-ca-cert\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522396 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522353 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-podinfo\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522629 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522397 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-envoy\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522629 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522426 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-credential-socket\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522629 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522432 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:59:08.522629 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522494 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-certs\") pod \"6e52a964-68ef-4b53-b13d-bec6ae1832da\" (UID: \"6e52a964-68ef-4b53-b13d-bec6ae1832da\") " Jul 29 19:59:08.522856 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522719 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e52a964-68ef-4b53-b13d-bec6ae1832da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:59:08.523018 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522976 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:59:08.523018 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.522996 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:59:08.523224 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.523055 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:08.523224 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.523079 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e52a964-68ef-4b53-b13d-bec6ae1832da-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:08.523346 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.523286 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:59:08.524479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.524453 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-kube-api-access-pgwpn" (OuterVolumeSpecName: "kube-api-access-pgwpn") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "kube-api-access-pgwpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:59:08.524621 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.524577 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:59:08.524621 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.524591 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:59:08.524621 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.524591 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e52a964-68ef-4b53-b13d-bec6ae1832da" (UID: "6e52a964-68ef-4b53-b13d-bec6ae1832da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:59:08.623924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.623885 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:08.623924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.623920 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:08.623924 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.623930 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:08.624176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.623939 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:08.624176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.623948 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgwpn\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-kube-api-access-pgwpn\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:08.624176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.623957 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:08.624176 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:08.623965 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e52a964-68ef-4b53-b13d-bec6ae1832da-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\"" Jul 29 19:59:09.106119 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.106081 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kc6j5_27c6a3fc-aff8-4c5f-a394-959078adbce1/manager/0.log" Jul 29 19:59:09.170666 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.170631 2575 generic.go:356] "Generic (PLEG): container finished" podID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerID="d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015" exitCode=0 Jul 29 19:59:09.170850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.170708 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" Jul 29 19:59:09.170850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.170722 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" event={"ID":"6e52a964-68ef-4b53-b13d-bec6ae1832da","Type":"ContainerDied","Data":"d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015"} Jul 29 19:59:09.170850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.170806 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl" event={"ID":"6e52a964-68ef-4b53-b13d-bec6ae1832da","Type":"ContainerDied","Data":"a105337dd315b1fcc9b446dab356391c970994583359ca13ebc0666c03645e16"} Jul 29 19:59:09.170850 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.170832 2575 scope.go:117] "RemoveContainer" containerID="d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015" Jul 29 19:59:09.181865 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.181848 2575 scope.go:117] "RemoveContainer" containerID="9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3" Jul 29 19:59:09.190417 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.190395 2575 scope.go:117] "RemoveContainer" containerID="d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015" Jul 29 19:59:09.190840 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:59:09.190810 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015\": container with ID starting with d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015 not found: ID does not exist" containerID="d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015" Jul 29 19:59:09.190984 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.190851 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015"} err="failed to get container status \"d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015\": rpc error: code = NotFound desc = could not find container \"d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015\": container with ID starting with d41a1a6e6245ffa521e4c239e3f8d62f68fc78dfb7d3417d4eda975c12db7015 not found: ID does not exist" Jul 29 19:59:09.190984 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.190895 2575 scope.go:117] "RemoveContainer" containerID="9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3" Jul 29 19:59:09.191189 ip-10-0-142-16 kubenswrapper[2575]: E0729 19:59:09.191169 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3\": container with ID starting with 9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3 not found: ID does not exist" containerID="9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3" Jul 29 19:59:09.191252 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.191195 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3"} err="failed to get container status \"9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3\": rpc error: code = NotFound desc = could not find container \"9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3\": container with ID starting with 9c4b435f9a5066bcfff152f355c299fe4647c996514e8dca4554ce88310d8db3 not found: ID does not exist" Jul 29 19:59:09.193903 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.193881 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl"] Jul 29 19:59:09.197540 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.197501 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-808386-openshift-default-6476498c4d-c9wjl"] Jul 29 19:59:09.214889 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.214858 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v5h7t_e537131c-098c-4ecc-a4f0-cf0b25d71930/manager/0.log" Jul 29 19:59:09.334245 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.334217 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c6f6796c6-74wqj_b03c7525-db2b-41d8-b372-7cee498fb6ba/manager/0.log" Jul 29 19:59:09.448115 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.448090 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-czrkh_870f2696-fc72-4ad2-93b8-756d37203175/server/0.log" Jul 29 19:59:09.549954 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.549920 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" path="/var/lib/kubelet/pods/6e52a964-68ef-4b53-b13d-bec6ae1832da/volumes" Jul 29 19:59:09.573184 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.573150 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vz6ht_cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8/manager/0.log" Jul 29 19:59:09.687987 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.687956 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b7c5ccf8f-47hp9_20f0196c-5ad4-450c-858a-89aa31d08a61/copy-manifests/0.log" Jul 29 19:59:09.700412 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:09.700327 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b7c5ccf8f-47hp9_20f0196c-5ad4-450c-858a-89aa31d08a61/manager/0.log" Jul 29 19:59:10.366474 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:10.366428 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85cdb98bf6-wv698_723506d5-645a-48c8-96c2-4acd2bb8e126/maas-api/0.log" Jul 29 19:59:10.837406 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:10.837375 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jc9w4_f449a28f-feef-4724-82d8-cb9ec4d19279/postgres/0.log" Jul 29 19:59:11.599731 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.599700 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g_807cf65d-fbcf-4cfd-bfe8-6154f32fc057/util/0.log" Jul 29 19:59:11.606198 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.606178 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g_807cf65d-fbcf-4cfd-bfe8-6154f32fc057/pull/0.log" Jul 29 19:59:11.613068 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.613043 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g_807cf65d-fbcf-4cfd-bfe8-6154f32fc057/extract/0.log" Jul 29 19:59:11.725453 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.725426 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp_0c07a5c0-84a6-4145-88e4-81464dca4c60/util/0.log" Jul 29 19:59:11.730966 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.730944 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp_0c07a5c0-84a6-4145-88e4-81464dca4c60/pull/0.log" Jul 29 19:59:11.736139 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.736120 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp_0c07a5c0-84a6-4145-88e4-81464dca4c60/extract/0.log" Jul 29 19:59:11.846364 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.846333 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9_875ad67a-bb82-48fa-8e7d-671d1162e40a/util/0.log" Jul 29 19:59:11.852837 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.852777 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9_875ad67a-bb82-48fa-8e7d-671d1162e40a/pull/0.log" Jul 29 19:59:11.859350 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.859324 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9_875ad67a-bb82-48fa-8e7d-671d1162e40a/extract/0.log" Jul 29 19:59:11.976045 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.976016 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl_a0f9c84d-c11b-4e84-9393-3a072c48258a/util/0.log" Jul 29 19:59:11.982880 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.982854 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl_a0f9c84d-c11b-4e84-9393-3a072c48258a/pull/0.log" Jul 29 19:59:11.989055 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:11.989035 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl_a0f9c84d-c11b-4e84-9393-3a072c48258a/extract/0.log" Jul 29 19:59:12.101740 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:12.101701 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-565f6d5884-w59bh_e36391c1-e0a0-441a-9392-9b73db31f7ab/authorino/0.log" Jul 29 19:59:12.218931 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:12.218903 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-z2vvl_ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e/manager/0.log" Jul 29 19:59:12.430420 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:12.430391 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zfxwn_dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764/kuadrant-console-plugin/0.log" Jul 29 19:59:12.547015 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:12.546922 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-qg4sz_2caba4c5-0bf0-41b5-9c06-62356c6aee1e/registry-server/0.log" Jul 29 19:59:12.657155 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:12.657128 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2vj82_ea2c139c-5793-4283-8dee-60b33a8136eb/manager/0.log" Jul 29 19:59:13.215010 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:13.214982 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9_9eba4d7d-63f4-49a4-b90b-02c7888fc0c1/istio-proxy/0.log" Jul 29 19:59:13.436836 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:13.436803 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f5845fd56-ht8ts_89bcf019-aa18-43e1-a8ad-81fe99025797/kube-auth-proxy/0.log" Jul 29 19:59:13.656688 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:13.656654 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fdsmh_2f48062d-30b4-4a4b-ac41-3025ba43aad9/istio-proxy/1.log" Jul 29 19:59:13.767270 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:13.767233 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2vdwb_265893ee-6692-4d22-9d62-b4749ebc6b0b/payload-pre-processing/0.log" Jul 29 19:59:14.323760 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:14.323727 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wcgcd_3c290c0c-a853-48c5-bf92-c5b0b01583af/main/0.log" Jul 29 19:59:14.877984 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:14.877955 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-9k2nl_f84d1fa4-d0a3-4794-8e67-8f028d00724e/main/0.log" Jul 29 19:59:23.639503 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:23.639458 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xgvm6_40076bf8-469e-4e66-a9ba-85e5a8e876b5/global-pull-secret-syncer/0.log" Jul 29 19:59:23.744479 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:23.744423 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hj2k8_a3f27f0f-faaa-4939-8867-64954f08e3d1/konnectivity-agent/0.log" Jul 29 19:59:23.873944 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:23.873917 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-16.ec2.internal_bf51505803dd721bf45ed7e8f52c4dcf/haproxy/0.log" Jul 29 19:59:29.002690 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.002643 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g_807cf65d-fbcf-4cfd-bfe8-6154f32fc057/extract/0.log" Jul 29 19:59:29.027713 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.027689 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g_807cf65d-fbcf-4cfd-bfe8-6154f32fc057/util/0.log" Jul 29 19:59:29.050530 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.050495 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759x2w9g_807cf65d-fbcf-4cfd-bfe8-6154f32fc057/pull/0.log" Jul 29 19:59:29.082600 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.082565 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp_0c07a5c0-84a6-4145-88e4-81464dca4c60/extract/0.log" Jul 29 19:59:29.110063 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.110035 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp_0c07a5c0-84a6-4145-88e4-81464dca4c60/util/0.log" Jul 29 19:59:29.129622 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.129599 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bhqpp_0c07a5c0-84a6-4145-88e4-81464dca4c60/pull/0.log" Jul 29 19:59:29.162952 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.162928 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9_875ad67a-bb82-48fa-8e7d-671d1162e40a/extract/0.log" Jul 29 19:59:29.187308 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.187280 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9_875ad67a-bb82-48fa-8e7d-671d1162e40a/util/0.log" Jul 29 19:59:29.211993 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.211958 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v8sp9_875ad67a-bb82-48fa-8e7d-671d1162e40a/pull/0.log" Jul 29 19:59:29.248121 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.248089 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl_a0f9c84d-c11b-4e84-9393-3a072c48258a/extract/0.log" Jul 29 19:59:29.281278 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.281206 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl_a0f9c84d-c11b-4e84-9393-3a072c48258a/util/0.log" Jul 29 19:59:29.306894 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.306867 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17zwzl_a0f9c84d-c11b-4e84-9393-3a072c48258a/pull/0.log" Jul 29 19:59:29.668191 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.668142 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-565f6d5884-w59bh_e36391c1-e0a0-441a-9392-9b73db31f7ab/authorino/0.log" Jul 29 19:59:29.706865 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.706840 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-z2vvl_ad186fd6-2d4c-41e4-9d6b-1a0922a12f2e/manager/0.log" Jul 29 19:59:29.767949 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.767927 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zfxwn_dfbd83d9-4ccb-4dd1-82fd-3ce127e4f764/kuadrant-console-plugin/0.log" Jul 29 19:59:29.812950 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.812916 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-qg4sz_2caba4c5-0bf0-41b5-9c06-62356c6aee1e/registry-server/0.log" Jul 29 19:59:29.957873 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:29.957789 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2vj82_ea2c139c-5793-4283-8dee-60b33a8136eb/manager/0.log" Jul 29 19:59:31.773211 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:31.773174 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_569c6b31-d722-4187-868b-bc4a6f019c44/alertmanager/0.log" Jul 29 19:59:31.797153 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:31.797131 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_569c6b31-d722-4187-868b-bc4a6f019c44/config-reloader/0.log" Jul 29 19:59:31.819588 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:31.819562 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_569c6b31-d722-4187-868b-bc4a6f019c44/kube-rbac-proxy-web/0.log" Jul 29 19:59:31.843417 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:31.843386 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_569c6b31-d722-4187-868b-bc4a6f019c44/kube-rbac-proxy/0.log" Jul 29 19:59:31.866780 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:31.866740 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_569c6b31-d722-4187-868b-bc4a6f019c44/kube-rbac-proxy-metric/0.log" Jul 29 19:59:31.890832 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:31.890809 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_569c6b31-d722-4187-868b-bc4a6f019c44/prom-label-proxy/0.log" Jul 29 19:59:31.915708 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:31.915662 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_569c6b31-d722-4187-868b-bc4a6f019c44/init-config-reloader/0.log" Jul 29 19:59:32.216292 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.216265 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chrfj_833b3fa9-8636-4eda-a9d4-543a83510835/node-exporter/0.log" Jul 29 19:59:32.241784 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.241755 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chrfj_833b3fa9-8636-4eda-a9d4-543a83510835/kube-rbac-proxy/0.log" Jul 29 19:59:32.262724 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.262699 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chrfj_833b3fa9-8636-4eda-a9d4-543a83510835/init-textfile/0.log" Jul 29 19:59:32.723967 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.723936 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bbd786858-blfw8_f5edd272-19b9-440e-a5b1-a2ae119d11af/telemeter-client/0.log" Jul 29 19:59:32.749870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.749831 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bbd786858-blfw8_f5edd272-19b9-440e-a5b1-a2ae119d11af/reload/0.log" Jul 29 19:59:32.771102 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.771069 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bbd786858-blfw8_f5edd272-19b9-440e-a5b1-a2ae119d11af/kube-rbac-proxy/0.log" Jul 29 19:59:32.802973 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.802949 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84447cf48b-z2d69_3bed8f20-6b8a-4a45-82ad-313c6c9ff53e/thanos-query/0.log" Jul 29 19:59:32.824719 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.824693 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84447cf48b-z2d69_3bed8f20-6b8a-4a45-82ad-313c6c9ff53e/kube-rbac-proxy-web/0.log" Jul 29 19:59:32.868974 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.868949 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84447cf48b-z2d69_3bed8f20-6b8a-4a45-82ad-313c6c9ff53e/kube-rbac-proxy/0.log" Jul 29 19:59:32.890355 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.890336 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84447cf48b-z2d69_3bed8f20-6b8a-4a45-82ad-313c6c9ff53e/prom-label-proxy/0.log" Jul 29 19:59:32.911460 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.911441 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84447cf48b-z2d69_3bed8f20-6b8a-4a45-82ad-313c6c9ff53e/kube-rbac-proxy-rules/0.log" Jul 29 19:59:32.933326 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:32.933304 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84447cf48b-z2d69_3bed8f20-6b8a-4a45-82ad-313c6c9ff53e/kube-rbac-proxy-metrics/0.log" Jul 29 19:59:33.522397 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522366 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh"] Jul 29 19:59:33.522822 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522809 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" Jul 29 19:59:33.522870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522825 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" Jul 29 19:59:33.522870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522836 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" Jul 29 19:59:33.522870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522842 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" Jul 29 19:59:33.522870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522856 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" Jul 29 19:59:33.522870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522861 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522872 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522878 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522889 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522894 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522961 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522969 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7ea91b-87cc-435a-887e-a542be0409d5" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522977 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522984 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522990 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a77f29-d9ee-49e0-8616-59e6b902227d" containerName="istio-proxy" Jul 29 19:59:33.523029 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.522996 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e52a964-68ef-4b53-b13d-bec6ae1832da" containerName="istio-proxy" Jul 29 19:59:33.527833 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.527815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.530559 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.530535 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxqqs\"/\"openshift-service-ca.crt\"" Jul 29 19:59:33.530742 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.530719 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kxqqs\"/\"kube-root-ca.crt\"" Jul 29 19:59:33.531550 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.531510 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kxqqs\"/\"default-dockercfg-tgndr\"" Jul 29 19:59:33.533709 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.533687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh"] Jul 29 19:59:33.538523 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.538489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-proc\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.538615 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.538565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-sys\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.538615 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.538604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-lib-modules\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.538700 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.538668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrwgp\" (UniqueName: \"kubernetes.io/projected/f3cb4972-973d-4f83-885d-b71cbac2b225-kube-api-access-jrwgp\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.538736 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.538719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-podres\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639596 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrwgp\" (UniqueName: \"kubernetes.io/projected/f3cb4972-973d-4f83-885d-b71cbac2b225-kube-api-access-jrwgp\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639797 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-podres\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639797 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-proc\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639797 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-sys\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639797 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-lib-modules\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639979 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-podres\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639979 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639831 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-sys\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639979 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-proc\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.639979 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.639870 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3cb4972-973d-4f83-885d-b71cbac2b225-lib-modules\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.649149 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.649128 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrwgp\" (UniqueName: \"kubernetes.io/projected/f3cb4972-973d-4f83-885d-b71cbac2b225-kube-api-access-jrwgp\") pod \"perf-node-gather-daemonset-2hjqh\" (UID: \"f3cb4972-973d-4f83-885d-b71cbac2b225\") " pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.839620 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.839531 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:33.977020 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:33.976995 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh"] Jul 29 19:59:33.978838 ip-10-0-142-16 kubenswrapper[2575]: W0729 19:59:33.978812 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3cb4972_973d_4f83_885d_b71cbac2b225.slice/crio-c6a13425a9e3df0a1bdc8941f8e745cd41872bfca73320e731f909e7c2ca552c WatchSource:0}: Error finding container c6a13425a9e3df0a1bdc8941f8e745cd41872bfca73320e731f909e7c2ca552c: Status 404 returned error can't find the container with id c6a13425a9e3df0a1bdc8941f8e745cd41872bfca73320e731f909e7c2ca552c Jul 29 19:59:34.177802 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:34.177762 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-fmxls_54be0aba-6d57-43bb-a21e-0f458e2b38fd/networking-console-plugin/0.log" Jul 29 19:59:34.319596 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:34.319562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" event={"ID":"f3cb4972-973d-4f83-885d-b71cbac2b225","Type":"ContainerStarted","Data":"4a4bb50da0f32061a82a96c653f98b1908808050c90af967c9b26c97d5766308"} Jul 29 19:59:34.319870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:34.319613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" event={"ID":"f3cb4972-973d-4f83-885d-b71cbac2b225","Type":"ContainerStarted","Data":"c6a13425a9e3df0a1bdc8941f8e745cd41872bfca73320e731f909e7c2ca552c"} Jul 29 19:59:34.319870 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:34.319656 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:34.336020 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:34.335966 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" podStartSLOduration=1.335952274 podStartE2EDuration="1.335952274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:34.334574519 +0000 UTC m=+3555.351595616" watchObservedRunningTime="2026-07-29 19:59:34.335952274 +0000 UTC m=+3555.352973370" Jul 29 19:59:37.014963 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:37.014935 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zljtc_84f87992-9daa-4f03-853e-906e0268c139/dns/0.log" Jul 29 19:59:37.038351 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:37.038326 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zljtc_84f87992-9daa-4f03-853e-906e0268c139/kube-rbac-proxy/0.log" Jul 29 19:59:37.064597 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:37.064563 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4zjgp_b8eb708d-e411-4fc8-bca7-9781b260c738/dns-node-resolver/0.log" Jul 29 19:59:37.736588 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:37.736545 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qxrp2_7d1c8b71-e806-48a9-9037-5fe23acc92ec/node-ca/0.log" Jul 29 19:59:38.761754 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:38.761723 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9_9eba4d7d-63f4-49a4-b90b-02c7888fc0c1/istio-proxy/0.log" Jul 29 19:59:38.932738 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:38.932705 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f5845fd56-ht8ts_89bcf019-aa18-43e1-a8ad-81fe99025797/kube-auth-proxy/0.log" Jul 29 19:59:38.990219 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:38.990189 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fdsmh_2f48062d-30b4-4a4b-ac41-3025ba43aad9/istio-proxy/0.log" Jul 29 19:59:39.016576 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:39.016491 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fdsmh_2f48062d-30b4-4a4b-ac41-3025ba43aad9/istio-proxy/1.log" Jul 29 19:59:39.046984 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:39.046956 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2vdwb_265893ee-6692-4d22-9d62-b4749ebc6b0b/payload-pre-processing/0.log" Jul 29 19:59:39.994937 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:39.994905 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wpq2w_ba24905b-57b4-40fd-ad0f-4ca06010d4d5/serve-healthcheck-canary/0.log" Jul 29 19:59:40.336728 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:40.336650 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kxqqs/perf-node-gather-daemonset-2hjqh" Jul 29 19:59:40.515591 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:40.515560 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llzvq_ebd00696-c986-47ae-a7d3-34469a66ed98/kube-rbac-proxy/0.log" Jul 29 19:59:40.537771 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:40.537747 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llzvq_ebd00696-c986-47ae-a7d3-34469a66ed98/exporter/0.log" Jul 29 19:59:40.564187 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:40.564161 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llzvq_ebd00696-c986-47ae-a7d3-34469a66ed98/extractor/0.log" Jul 29 19:59:43.086812 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:43.086784 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kc6j5_27c6a3fc-aff8-4c5f-a394-959078adbce1/manager/0.log" Jul 29 19:59:43.301387 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:43.301352 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v5h7t_e537131c-098c-4ecc-a4f0-cf0b25d71930/manager/0.log" Jul 29 19:59:43.450724 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:43.446846 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c6f6796c6-74wqj_b03c7525-db2b-41d8-b372-7cee498fb6ba/manager/0.log" Jul 29 19:59:43.471005 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:43.470971 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-czrkh_870f2696-fc72-4ad2-93b8-756d37203175/server/0.log" Jul 29 19:59:43.546493 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:43.546463 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vz6ht_cf41e7f5-aca7-44cd-a4f6-66c1e08fe1c8/manager/0.log" Jul 29 19:59:43.572357 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:43.572325 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b7c5ccf8f-47hp9_20f0196c-5ad4-450c-858a-89aa31d08a61/manager/0.log" Jul 29 19:59:43.599228 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:43.599199 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b7c5ccf8f-47hp9_20f0196c-5ad4-450c-858a-89aa31d08a61/copy-manifests/0.log" Jul 29 19:59:45.076574 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:45.076482 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9_9eba4d7d-63f4-49a4-b90b-02c7888fc0c1/istio-proxy/0.log" Jul 29 19:59:45.271814 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:45.271789 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f5845fd56-ht8ts_89bcf019-aa18-43e1-a8ad-81fe99025797/kube-auth-proxy/0.log" Jul 29 19:59:45.317989 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:45.317958 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fdsmh_2f48062d-30b4-4a4b-ac41-3025ba43aad9/istio-proxy/0.log" Jul 29 19:59:45.347408 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:45.347332 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fdsmh_2f48062d-30b4-4a4b-ac41-3025ba43aad9/istio-proxy/1.log" Jul 29 19:59:45.367314 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:45.367286 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2vdwb_265893ee-6692-4d22-9d62-b4749ebc6b0b/payload-pre-processing/0.log" Jul 29 19:59:46.804443 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:46.804414 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cpzlb9_9eba4d7d-63f4-49a4-b90b-02c7888fc0c1/istio-proxy/0.log" Jul 29 19:59:46.955217 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:46.955191 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f5845fd56-ht8ts_89bcf019-aa18-43e1-a8ad-81fe99025797/kube-auth-proxy/0.log" Jul 29 19:59:47.009355 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:47.009327 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fdsmh_2f48062d-30b4-4a4b-ac41-3025ba43aad9/istio-proxy/0.log" Jul 29 19:59:47.034961 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:47.034932 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fdsmh_2f48062d-30b4-4a4b-ac41-3025ba43aad9/istio-proxy/1.log" Jul 29 19:59:47.061211 ip-10-0-142-16 kubenswrapper[2575]: I0729 19:59:47.061122 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2vdwb_265893ee-6692-4d22-9d62-b4749ebc6b0b/payload-pre-processing/0.log"