Jul 27 16:56:43.410426 ip-10-0-128-46 systemd[1]: Starting Kubernetes Kubelet... Jul 27 16:56:43.788121 ip-10-0-128-46 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 16:56:43.788121 ip-10-0-128-46 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 16:56:43.788121 ip-10-0-128-46 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 16:56:43.788121 ip-10-0-128-46 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 16:56:43.788121 ip-10-0-128-46 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 16:56:43.789812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.789706 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 16:56:43.793990 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.793974 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:56:43.793990 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.793989 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.793997 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794001 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794012 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794016 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794020 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794022 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794025 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794027 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794030 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794033 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794036 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794038 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794041 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794043 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794046 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794049 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794052 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794054 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:56:43.794057 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794057 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794060 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794062 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794065 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794069 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794072 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794075 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794077 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794080 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794082 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794085 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794087 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794090 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794094 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794096 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794099 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794101 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794104 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794107 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794109 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:56:43.794522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794111 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794114 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794116 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794119 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794121 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794124 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794126 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794128 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794131 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794133 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794136 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794138 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794141 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794143 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794147 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794149 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794152 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794155 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794157 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794160 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:56:43.795002 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794162 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794165 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794167 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794170 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794184 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794187 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794190 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794192 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794195 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794198 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794201 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794205 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794208 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794210 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794213 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794215 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794218 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794220 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794223 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794227 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:56:43.795511 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794231 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794234 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794237 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794239 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794242 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794244 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794248 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794251 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794254 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794256 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794259 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794262 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794264 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794267 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794270 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794273 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794277 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794280 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794283 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:56:43.796039 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794960 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794966 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794970 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794973 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794976 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794980 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794983 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794985 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794988 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794990 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794993 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794996 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.794999 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795001 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795004 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795006 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795009 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795012 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795015 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795018 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:56:43.796522 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795021 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795024 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795026 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795029 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795032 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795034 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795037 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795039 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795042 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795045 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795049 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795052 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795055 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795057 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795059 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795062 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795065 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795068 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795071 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:56:43.797024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795073 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795078 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795081 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795084 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795772 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795778 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795782 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795785 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795788 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795791 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795795 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795798 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795801 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795804 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795807 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795810 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795812 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795815 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795818 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:56:43.797503 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795821 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795823 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795826 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795828 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795831 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795833 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795836 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795838 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795842 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795844 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795847 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795849 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795853 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795855 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795858 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795860 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795863 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795866 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795868 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795871 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:56:43.797968 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795873 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795875 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795878 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795881 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795883 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795886 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795889 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795892 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795894 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795897 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795899 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795902 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795904 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795907 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795909 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795912 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795915 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795917 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795920 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795922 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:56:43.798489 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.795925 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796764 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796771 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796774 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796778 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796781 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796784 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796787 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796789 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796792 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796795 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796798 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796801 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796803 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796806 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796808 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796811 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796813 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796816 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796818 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:56:43.798988 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796821 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796823 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796826 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796830 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796834 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796837 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796840 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796843 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796846 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796848 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796851 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796854 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796857 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796860 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796863 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796866 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796868 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796871 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796874 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:56:43.799488 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796877 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796879 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796882 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796884 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796887 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796890 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796892 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796896 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796900 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796902 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796905 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796907 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796910 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796912 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796915 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796917 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796920 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796922 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796925 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:56:43.799952 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796927 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796930 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796932 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796935 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796937 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796940 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796942 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796945 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796948 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796951 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796953 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796956 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796959 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796961 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796963 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796966 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796969 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796972 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796974 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796976 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:56:43.800469 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796980 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796983 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796985 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796988 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796990 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796992 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796995 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.796997 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797000 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797002 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797005 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797007 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797010 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797012 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797015 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797018 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797020 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797022 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797025 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797028 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:56:43.800969 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797030 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797033 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797102 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797111 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797120 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797127 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797138 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797142 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797147 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797151 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797154 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797157 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797161 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797164 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797167 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797170 2557 flags.go:64] FLAG: --cgroup-root="" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797183 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797187 2557 flags.go:64] FLAG: --client-ca-file="" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797190 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797193 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797198 2557 flags.go:64] FLAG: --cluster-domain="" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797200 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797204 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797207 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 16:56:43.801475 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797211 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797215 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797218 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797221 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797225 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797228 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797231 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797234 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797237 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797240 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797245 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797248 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797251 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797254 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797257 2557 flags.go:64] FLAG: --enable-server="true" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797263 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797267 2557 flags.go:64] FLAG: --event-burst="100" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797270 2557 flags.go:64] FLAG: --event-qps="50" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797274 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797277 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797280 2557 flags.go:64] FLAG: --eviction-hard="" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797284 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797287 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797290 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797293 2557 flags.go:64] FLAG: --eviction-soft="" Jul 27 16:56:43.802056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797295 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797299 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797301 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797304 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797307 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797310 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797313 2557 flags.go:64] FLAG: --feature-gates="" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797317 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797320 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797323 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797326 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797329 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797332 2557 flags.go:64] FLAG: --help="false" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797336 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-128-46.ec2.internal" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797339 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797342 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797345 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797348 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797352 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797355 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797358 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797361 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797367 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 16:56:43.802670 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797371 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797374 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797377 2557 flags.go:64] FLAG: --kube-reserved="" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797380 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797383 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797386 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797389 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797392 2557 flags.go:64] FLAG: --lock-file="" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797394 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797398 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797401 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797406 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797410 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797413 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797416 2557 flags.go:64] FLAG: --logging-format="text" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797419 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797422 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797425 2557 flags.go:64] FLAG: --manifest-url="" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797428 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797432 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797435 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797439 2557 flags.go:64] FLAG: --max-pods="110" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797442 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797445 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797447 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 16:56:43.803295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797450 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797453 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797456 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797459 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797466 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797469 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797473 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797477 2557 flags.go:64] FLAG: --pod-cidr="" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797479 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797485 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797487 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797490 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797493 2557 flags.go:64] FLAG: --port="10250" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797496 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797499 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a582a2ccbbd7ba5e" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797502 2557 flags.go:64] FLAG: --qos-reserved="" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797506 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797509 2557 flags.go:64] FLAG: --register-node="true" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797512 2557 flags.go:64] FLAG: --register-with-taints="" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797518 2557 flags.go:64] FLAG: --registry-burst="10" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797521 2557 flags.go:64] FLAG: --registry-qps="5" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797524 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797527 2557 flags.go:64] FLAG: --reserved-memory="" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797530 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797533 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 16:56:43.804208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797536 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797539 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797542 2557 flags.go:64] FLAG: --runonce="false" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797545 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797548 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797551 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797553 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797556 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797559 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797562 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797565 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797568 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797571 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797573 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797577 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797581 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797583 2557 flags.go:64] FLAG: --system-cgroups="" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797586 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797591 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797594 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797597 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797601 2557 flags.go:64] FLAG: --tls-min-version="" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797605 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797607 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797610 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 16:56:43.804994 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797613 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797617 2557 flags.go:64] FLAG: --v="2" Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797621 2557 flags.go:64] FLAG: --version="false" Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797625 2557 flags.go:64] FLAG: --vmodule="" Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797628 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.797631 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797752 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797756 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797759 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797762 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797766 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797770 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797773 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797776 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797779 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797781 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797783 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797786 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797789 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797791 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797794 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:56:43.805640 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797796 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797799 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797802 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797804 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797807 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797809 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797812 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797814 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797817 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797820 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797822 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797825 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797829 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797832 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797834 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797837 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797839 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797842 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797845 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797847 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:56:43.806141 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797850 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797852 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797855 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797857 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797860 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797862 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797865 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797869 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797872 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797875 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797878 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797881 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797883 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797886 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797889 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797892 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797894 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797897 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797900 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:56:43.806644 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797902 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797905 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797907 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797910 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797912 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797916 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797920 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797923 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797926 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797930 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797932 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797935 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797937 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797940 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797943 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797946 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797949 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797952 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797954 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797957 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:56:43.807105 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797960 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797962 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797965 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797967 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797970 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797973 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797976 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797979 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797981 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797984 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797987 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797989 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797992 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797994 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797997 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.797999 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798002 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798005 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798008 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798010 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:56:43.807604 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798013 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798015 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798018 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798020 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.798022 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.798028 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.806307 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.806323 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.807007 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.807014 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807070 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807076 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807079 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807082 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:56:43.808086 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807086 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807088 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807091 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807094 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807097 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807099 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807102 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807104 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807107 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807110 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807113 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807115 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807118 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807123 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807137 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807140 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807144 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807147 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807150 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:56:43.808549 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807153 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807156 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807158 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807161 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807163 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807167 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807171 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807188 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807191 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807194 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807197 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807200 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807202 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807205 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807208 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807211 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807213 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807216 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807218 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807221 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:56:43.809007 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807224 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807227 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807230 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807233 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807236 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807238 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807241 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807243 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807246 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807248 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807251 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807254 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807257 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807259 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807262 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807264 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807267 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807269 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807273 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807276 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:56:43.809527 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807278 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807281 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807284 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807287 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807291 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807295 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807298 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807301 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807304 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807308 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807312 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807314 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807317 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807320 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807322 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807325 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807329 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807331 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807334 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807338 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:56:43.810024 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807340 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807343 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807346 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807349 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807351 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807354 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807356 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807359 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807361 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807364 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807367 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807370 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807373 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807375 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807378 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:56:43.810528 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807380 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.807385 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807502 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807507 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807510 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807514 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807516 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807519 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807522 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807524 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807527 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807529 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807532 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807535 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807538 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 16:56:43.810891 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807540 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807543 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807546 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807548 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807551 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807554 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807557 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807561 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807566 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807570 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807573 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807575 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807579 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807581 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807584 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807587 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807590 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807592 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807595 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 16:56:43.811281 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807597 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807600 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807602 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807605 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807607 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807610 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807612 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807615 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807618 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807620 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807623 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807625 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807628 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807631 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807634 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807636 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807639 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807642 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807644 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807647 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 16:56:43.811743 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807649 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807652 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807654 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807657 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807659 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807662 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807665 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807667 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807670 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807672 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807675 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807677 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807679 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807682 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807684 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807688 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807691 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807694 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807697 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807700 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 16:56:43.812253 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807702 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807705 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807707 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807710 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807712 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807715 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807718 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807720 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807723 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807725 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807728 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807731 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807733 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807736 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807738 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807741 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807743 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807746 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807748 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 16:56:43.812724 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807751 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807753 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807756 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807758 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807761 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807764 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807766 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:43.807768 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.807773 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.807878 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.809896 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.811250 2557 server.go:1019] "Starting client certificate rotation" Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.811353 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 16:56:43.813205 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.812018 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 16:56:43.813570 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.813539 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 16:56:43.817946 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.817931 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 16:56:43.830324 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.830308 2557 log.go:25] "Validated CRI v1 runtime API" Jul 27 16:56:43.832430 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.832414 2557 log.go:25] "Validated CRI v1 image API" Jul 27 16:56:43.833829 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.833810 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 16:56:43.836187 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.836152 2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 86fdc4ce-390c-4a0b-aaa1-7cf34e395920:/dev/nvme0n1p4 c632e602-40ff-4a5e-a8d8-61767b2950ac:/dev/nvme0n1p3] Jul 27 16:56:43.836256 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.836191 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 16:56:43.842268 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.842245 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 16:56:43.843215 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.843093 2557 manager.go:216] Machine: {Timestamp:2026-07-27 16:56:43.841656474 +0000 UTC m=+0.311982661 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099259 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e114eaf3ce84c081414082c6b027a SystemUUID:ec2e114e-af3c-e84c-0814-14082c6b027a BootID:f1862bd1-7fc2-4d29-bea1-b4e129d8fa52 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:89:ef:2e:15:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:89:ef:2e:15:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:bd:ec:a5:3a:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 16:56:43.843215 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.843211 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 16:56:43.843328 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.843289 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 16:56:43.845586 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.845559 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 16:56:43.845746 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.845590 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-46.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 16:56:43.846294 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.846282 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 16:56:43.846347 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.846299 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 16:56:43.846347 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.846309 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 16:56:43.847024 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.847013 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 16:56:43.847877 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.847865 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 16:56:43.849833 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.849819 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 27 16:56:43.850005 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.849993 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 16:56:43.850063 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.850034 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 27 16:56:43.850063 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.850054 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 16:56:43.850158 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.850074 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 16:56:43.850158 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.850088 2557 kubelet.go:398] "Adding apiserver pod source" Jul 27 16:56:43.850158 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.850101 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 16:56:43.850464 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.850450 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 16:56:43.850521 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.850473 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 16:56:43.852918 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.852887 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 16:56:43.854802 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.854788 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 16:56:43.854874 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.854817 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 16:56:43.856483 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856471 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 16:56:43.856529 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856488 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 16:56:43.856529 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856495 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 16:56:43.856529 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856501 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 16:56:43.856529 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856509 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 16:56:43.856529 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856514 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 16:56:43.856529 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856522 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 16:56:43.856529 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856527 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 16:56:43.856718 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856539 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 16:56:43.856718 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856546 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 16:56:43.856718 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856561 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 16:56:43.856718 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.856567 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 16:56:43.857248 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.857236 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 16:56:43.857282 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.857249 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 16:56:43.860623 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.860608 2557 server.go:1268] "Started kubelet" Jul 27 16:56:43.860751 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.860728 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 16:56:43.860851 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.860809 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 16:56:43.860910 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.860875 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 16:56:43.861095 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.861074 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-46.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 16:56:43.861229 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.861208 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-46.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 16:56:43.861299 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.861258 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 16:56:43.861513 ip-10-0-128-46 systemd[1]: Started Kubernetes Kubelet. Jul 27 16:56:43.863554 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.863541 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 27 16:56:43.865249 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.865232 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pbjxj" Jul 27 16:56:43.865359 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.865345 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 16:56:43.865827 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.865807 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 16:56:43.867858 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.866660 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-46.ec2.internal.18c633eefe4b963f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-46.ec2.internal,UID:ip-10-0-128-46.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-46.ec2.internal,},FirstTimestamp:2026-07-27 16:56:43.860579903 +0000 UTC m=+0.330906088,LastTimestamp:2026-07-27 16:56:43.860579903 +0000 UTC m=+0.330906088,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-46.ec2.internal,}" Jul 27 16:56:43.872046 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.872031 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pbjxj" Jul 27 16:56:43.873917 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.873902 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 16:56:43.873999 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.873953 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 16:56:43.874060 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.874020 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:43.874795 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.874623 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 16:56:43.874871 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.874821 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 16:56:43.874871 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.874835 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 16:56:43.875007 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.874992 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 27 16:56:43.875054 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.875008 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 27 16:56:43.875586 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.875565 2557 factory.go:55] Registering systemd factory Jul 27 16:56:43.875790 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.875774 2557 factory.go:223] Registration of the systemd container factory successfully Jul 27 16:56:43.876081 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.876067 2557 factory.go:153] Registering CRI-O factory Jul 27 16:56:43.876151 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.876084 2557 factory.go:223] Registration of the crio container factory successfully Jul 27 16:56:43.876151 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.876142 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 16:56:43.876262 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.876167 2557 factory.go:103] Registering Raw factory Jul 27 16:56:43.876262 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.876201 2557 manager.go:1196] Started watching for new ooms in manager Jul 27 16:56:43.876442 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.876415 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-46.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 16:56:43.876713 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.876687 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 16:56:43.876825 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.876806 2557 manager.go:349] Starting recovery of all containers Jul 27 16:56:43.887471 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.887449 2557 manager.go:354] Recovery completed Jul 27 16:56:43.891324 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.891220 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:56:43.893950 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.893932 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:56:43.894026 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.893964 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:56:43.894026 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.893980 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:56:43.894471 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.894458 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 16:56:43.894471 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.894469 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 16:56:43.894598 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.894490 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 27 16:56:43.894598 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.894471 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:56:43.895593 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.895579 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:56:43.895671 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.895605 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:56:43.895671 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.895621 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:56:43.896299 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.896284 2557 policy_none.go:49] "None policy: Start" Jul 27 16:56:43.896333 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.896307 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 16:56:43.896333 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.896323 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 16:56:43.897396 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.897385 2557 policy_none.go:47] "Start" Jul 27 16:56:43.929720 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.929703 2557 manager.go:337] "Starting Device Plugin manager" Jul 27 16:56:43.929796 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.929735 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 16:56:43.929796 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.929748 2557 server.go:87] "Starting device plugin registration server" Jul 27 16:56:43.929991 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.929978 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 16:56:43.930027 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.929992 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 16:56:43.930119 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.930105 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 16:56:43.930234 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.930204 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 16:56:43.930234 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:43.930216 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 16:56:43.930899 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.930875 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 16:56:43.930987 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:43.930936 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.030405 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.030373 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:56:44.032004 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.031940 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:56:44.032004 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.031974 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:56:44.032004 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.031985 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:56:44.032119 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.032013 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.041908 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.041861 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.041908 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.041891 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-46.ec2.internal\": node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.056436 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.056410 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 16:56:44.057871 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.057848 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 16:56:44.057959 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.057887 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 16:56:44.057959 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.057907 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 16:56:44.057959 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.057939 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 16:56:44.060207 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.060183 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.060847 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.060824 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:56:44.159139 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.159103 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal"] Jul 27 16:56:44.159329 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.159241 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:56:44.161113 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.161086 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.161501 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.161464 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:56:44.161613 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.161528 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:56:44.161613 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.161541 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:56:44.163047 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.163034 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:56:44.163268 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.163255 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.163319 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.163284 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:56:44.164952 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.164934 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:56:44.165048 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.164965 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:56:44.165048 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.164974 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:56:44.165512 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.165221 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:56:44.165512 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.165225 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:56:44.165512 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.165397 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:56:44.168030 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.168011 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.168122 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.168048 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 16:56:44.168838 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.168820 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientMemory" Jul 27 16:56:44.168903 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.168849 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 16:56:44.168903 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.168860 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeHasSufficientPID" Jul 27 16:56:44.176358 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.176339 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52b3e30e20b3d80b25b1dc1975a73979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal\" (UID: \"52b3e30e20b3d80b25b1dc1975a73979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.176427 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.176366 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b3e30e20b3d80b25b1dc1975a73979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal\" (UID: \"52b3e30e20b3d80b25b1dc1975a73979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.176427 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.176382 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e07fcea3b53729941e121c6d1709b9c1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-46.ec2.internal\" (UID: \"e07fcea3b53729941e121c6d1709b9c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.190961 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.190939 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-46.ec2.internal\" not found" node="ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.195555 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.195540 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-46.ec2.internal\" not found" node="ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.261188 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.261159 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.277345 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.277323 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b3e30e20b3d80b25b1dc1975a73979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal\" (UID: \"52b3e30e20b3d80b25b1dc1975a73979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.277410 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.277351 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e07fcea3b53729941e121c6d1709b9c1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-46.ec2.internal\" (UID: \"e07fcea3b53729941e121c6d1709b9c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.277410 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.277371 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52b3e30e20b3d80b25b1dc1975a73979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal\" (UID: \"52b3e30e20b3d80b25b1dc1975a73979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.277485 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.277438 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52b3e30e20b3d80b25b1dc1975a73979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal\" (UID: \"52b3e30e20b3d80b25b1dc1975a73979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.277485 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.277448 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b3e30e20b3d80b25b1dc1975a73979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal\" (UID: \"52b3e30e20b3d80b25b1dc1975a73979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.277485 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.277447 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e07fcea3b53729941e121c6d1709b9c1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-46.ec2.internal\" (UID: \"e07fcea3b53729941e121c6d1709b9c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.361737 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.361653 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.468769 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.468716 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.495129 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.495102 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.499279 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.499262 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.570827 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.570796 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.671729 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.671657 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.759894 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.759871 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:56:44.772062 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.772039 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.811235 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.811211 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 16:56:44.811682 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.811341 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 16:56:44.811682 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.811357 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 16:56:44.872298 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:44.872275 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-46.ec2.internal\" not found" Jul 27 16:56:44.879690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.879664 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 16:51:44 +0000 UTC" deadline="2028-02-09 22:02:33.857754415 +0000 UTC" Jul 27 16:56:44.879690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.879689 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13493h5m48.978069246s" Jul 27 16:56:44.964701 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.964647 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:56:44.974770 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.974751 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.987976 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.987959 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 16:56:44.989073 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.989059 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" Jul 27 16:56:44.999535 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:44.999519 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 16:56:45.075356 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.075336 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:56:45.302634 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:45.302599 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b3e30e20b3d80b25b1dc1975a73979.slice/crio-4a1b0ad5e8d66e0057d4bb18ed7cf3046bb71382f2eec796633317f746a0015f WatchSource:0}: Error finding container 4a1b0ad5e8d66e0057d4bb18ed7cf3046bb71382f2eec796633317f746a0015f: Status 404 returned error can't find the container with id 4a1b0ad5e8d66e0057d4bb18ed7cf3046bb71382f2eec796633317f746a0015f Jul 27 16:56:45.306722 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.306703 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 16:56:45.490343 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:45.490318 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07fcea3b53729941e121c6d1709b9c1.slice/crio-505720529e62cb2ef57dfda272695328dc10b58d09d740ccd4faa175a4093931 WatchSource:0}: Error finding container 505720529e62cb2ef57dfda272695328dc10b58d09d740ccd4faa175a4093931: Status 404 returned error can't find the container with id 505720529e62cb2ef57dfda272695328dc10b58d09d740ccd4faa175a4093931 Jul 27 16:56:45.852930 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.852901 2557 apiserver.go:52] "Watching apiserver" Jul 27 16:56:45.853573 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.852976 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:56:45.861754 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.861732 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 16:56:45.862224 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.862199 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw","openshift-cluster-node-tuning-operator/tuned-dpxmq","openshift-image-registry/node-ca-7gcp5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal","openshift-multus/multus-hvrq8","openshift-multus/network-metrics-daemon-4scq8","openshift-network-diagnostics/network-check-target-7xqlz","kube-system/global-pull-secret-syncer-t54n4","kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal","openshift-dns/node-resolver-wv5xc","openshift-multus/multus-additional-cni-plugins-rlcft","openshift-network-operator/iptables-alerter-2hfzd","openshift-ovn-kubernetes/ovnkube-node-ntgl8","kube-system/konnectivity-agent-jm5gm"] Jul 27 16:56:45.863721 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.863697 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.865520 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.865500 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.866922 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.866902 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.868479 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.868455 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:56:45.869492 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.869472 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 16:56:45.869596 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.869501 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9vw9d\"" Jul 27 16:56:45.869596 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.869509 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 16:56:45.869879 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.869852 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 16:56:45.870095 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.870072 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.870241 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.870117 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 16:56:45.870241 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.870215 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 16:56:45.870370 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.870264 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dsnbw\"" Jul 27 16:56:45.871521 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.871224 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 16:56:45.871521 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.871483 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 16:56:45.872035 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.872014 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 16:56:45.872189 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.872162 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 16:56:45.872292 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.872271 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4bpn6\"" Jul 27 16:56:45.872357 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.872023 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 16:56:45.873426 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.873387 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 16:56:45.873533 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.873476 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 16:56:45.874559 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.873387 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 16:56:45.874559 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.873784 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 16:56:45.874559 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.873387 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 16:56:45.874559 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.874055 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2z82r\"" Jul 27 16:56:45.874559 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.874426 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dlz5d\"" Jul 27 16:56:45.874833 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.874584 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:45.874833 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:45.874692 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:56:45.874833 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.874783 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 16:56:45.875361 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.875338 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 16:56:45.875723 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.875500 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 16:56:45.876922 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.876902 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.877018 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:45.876972 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:56:45.878714 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.878695 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:45.878793 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:45.878765 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:56:45.880304 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.880254 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.882546 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.882520 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:45.883679 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.883658 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5hxjn\"" Jul 27 16:56:45.883780 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.883719 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 16:56:45.883976 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.883953 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 16:56:45.884123 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.884103 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.884795 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.884777 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 16:56:45.884926 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.884884 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 16:56:45.885182 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.885165 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 16:56:45.885693 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.885672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:45.885803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.885719 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zqj6x\"" Jul 27 16:56:45.886383 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.886361 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-br6tc\"" Jul 27 16:56:45.886466 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.886397 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 16:56:45.886466 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.886437 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 16:56:45.887429 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887405 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9209f648-9b67-4793-aeb6-250869f631b9-host\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.887539 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887441 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-cnibin\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.887539 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887468 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-kubelet\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.887539 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887495 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjtb\" (UniqueName: \"kubernetes.io/projected/a555c7cc-d31b-47d5-9225-42a763aedc80-kube-api-access-txjtb\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.887539 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887519 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-ovn\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887553 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54w8b\" (UniqueName: \"kubernetes.io/projected/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-kube-api-access-54w8b\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887588 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-kubelet\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887603 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-systemd-units\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887617 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887635 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovnkube-config\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887661 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87e0c5f4-8b4b-4164-9d12-ebafec319642-hosts-file\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887682 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-cni-multus\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2402f312-cad4-44fa-a9ae-b752b73c6eb3-dbus\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887719 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-etc-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887735 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-node-log\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.887752 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887749 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-cni-bin\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887763 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4627z\" (UniqueName: \"kubernetes.io/projected/3f22a409-745d-4c35-9662-3c0f05e2b8c7-kube-api-access-4627z\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887778 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e0c5f4-8b4b-4164-9d12-ebafec319642-tmp-dir\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887792 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/675a1b63-8414-428b-a1ec-4084d018e04c-agent-certs\") pod \"konnectivity-agent-jm5gm\" (UID: \"675a1b63-8414-428b-a1ec-4084d018e04c\") " pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-cni-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887845 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-systemd\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887866 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9hs\" (UniqueName: \"kubernetes.io/projected/87e0c5f4-8b4b-4164-9d12-ebafec319642-kube-api-access-9l9hs\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887889 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-os-release\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887919 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a555c7cc-d31b-47d5-9225-42a763aedc80-cni-binary-copy\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887958 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-socket-dir-parent\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.887997 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-hostroot\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888032 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-daemon-config\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888064 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-multus-certs\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888090 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888153 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888204 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-env-overrides\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888232 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovn-node-metrics-cert\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.888344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888260 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-k8s-cni-cncf-io\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-netns\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888308 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-cni-bin\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888336 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-conf-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888378 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-run-netns\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888403 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovnkube-script-lib\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5ql\" (UniqueName: \"kubernetes.io/projected/9209f648-9b67-4793-aeb6-250869f631b9-kube-api-access-zf5ql\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888582 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/675a1b63-8414-428b-a1ec-4084d018e04c-konnectivity-ca\") pod \"konnectivity-agent-jm5gm\" (UID: \"675a1b63-8414-428b-a1ec-4084d018e04c\") " pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888595 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888605 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-etc-kubernetes\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888663 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2402f312-cad4-44fa-a9ae-b752b73c6eb3-kubelet-config\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888696 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-var-lib-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888715 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9209f648-9b67-4793-aeb6-250869f631b9-serviceca\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888739 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-system-cni-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888758 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888774 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-slash\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888789 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-log-socket\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888820 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-cni-netd\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.889185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.888966 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 16:56:45.889976 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.889314 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 16:56:45.889976 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.889473 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-55lfh\"" Jul 27 16:56:45.893118 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.893099 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 16:56:45.931728 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.931704 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2c7m6" Jul 27 16:56:45.943120 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.943102 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2c7m6" Jul 27 16:56:45.976299 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.976278 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 16:56:45.989124 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989098 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-cnibin\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.989263 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989139 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.989263 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989168 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-iptables-alerter-script\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:45.989263 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989229 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-run-netns\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.989426 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989260 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovnkube-script-lib\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.989426 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989286 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5ql\" (UniqueName: \"kubernetes.io/projected/9209f648-9b67-4793-aeb6-250869f631b9-kube-api-access-zf5ql\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.989426 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989310 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-run-netns\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.989426 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/675a1b63-8414-428b-a1ec-4084d018e04c-konnectivity-ca\") pod \"konnectivity-agent-jm5gm\" (UID: \"675a1b63-8414-428b-a1ec-4084d018e04c\") " pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:56:45.989426 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989360 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:45.989426 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989385 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysconfig\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.989426 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989407 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-kubernetes\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989431 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-lib-modules\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989459 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-var-lib-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989481 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-modprobe-d\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989503 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-systemd\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989542 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-cni-netd\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989571 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-cnibin\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989595 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn77p\" (UniqueName: \"kubernetes.io/projected/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-kube-api-access-kn77p\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989619 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-ovn\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989642 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-host\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989667 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989693 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-cni-multus\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2402f312-cad4-44fa-a9ae-b752b73c6eb3-dbus\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989774 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-socket-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:45.989806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989800 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-os-release\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989826 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989851 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td96j\" (UniqueName: \"kubernetes.io/projected/d057148b-ad2c-4737-b7b0-0938b54fdd93-kube-api-access-td96j\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989878 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-host-slash\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-node-log\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989929 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4627z\" (UniqueName: \"kubernetes.io/projected/3f22a409-745d-4c35-9662-3c0f05e2b8c7-kube-api-access-4627z\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989953 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/675a1b63-8414-428b-a1ec-4084d018e04c-konnectivity-ca\") pod \"konnectivity-agent-jm5gm\" (UID: \"675a1b63-8414-428b-a1ec-4084d018e04c\") " pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.989956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e0c5f4-8b4b-4164-9d12-ebafec319642-tmp-dir\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990003 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovnkube-script-lib\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990018 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-cni-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990043 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-hostroot\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-daemon-config\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990095 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-ovn\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990119 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-var-lib-kubelet\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990233 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e0c5f4-8b4b-4164-9d12-ebafec319642-tmp-dir\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990282 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-cni-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990457 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-hostroot\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.990527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-cni-netd\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990551 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-cnibin\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990600 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-cni-multus\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990602 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-var-lib-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990632 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-os-release\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990657 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990659 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-daemon-config\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990654 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-node-log\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990717 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-os-release\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990722 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2402f312-cad4-44fa-a9ae-b752b73c6eb3-dbus\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990711 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-device-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990769 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990801 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-k8s-cni-cncf-io\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990851 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-netns\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990926 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-netns\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.990973 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-k8s-cni-cncf-io\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991001 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-cni-bin\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.991371 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-conf-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991056 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-cni-bin\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991088 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991095 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-conf-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991122 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-sys-fs\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991236 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-cni-binary-copy\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2402f312-cad4-44fa-a9ae-b752b73c6eb3-kubelet-config\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991308 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysctl-d\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991331 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-run\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991359 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-registration-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9209f648-9b67-4793-aeb6-250869f631b9-serviceca\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991390 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2402f312-cad4-44fa-a9ae-b752b73c6eb3-kubelet-config\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991409 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-system-cni-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991435 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-slash\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991457 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-log-socket\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991479 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-slash\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991481 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9209f648-9b67-4793-aeb6-250869f631b9-host\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.992272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991502 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-system-cni-dir\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991509 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-kubelet\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991519 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9209f648-9b67-4793-aeb6-250869f631b9-host\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991543 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-var-lib-kubelet\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991577 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-log-socket\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991544 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjtb\" (UniqueName: \"kubernetes.io/projected/a555c7cc-d31b-47d5-9225-42a763aedc80-kube-api-access-txjtb\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991630 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991656 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54w8b\" (UniqueName: \"kubernetes.io/projected/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-kube-api-access-54w8b\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991698 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-system-cni-dir\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991726 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-kubelet\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:45.991761 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-systemd-units\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991790 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-kubelet\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991808 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991832 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-systemd-units\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:45.991836 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs podName:e328c2d0-1baf-4aba-8b4f-63c8accb0d29 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:46.491813198 +0000 UTC m=+2.962139393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs") pod "network-metrics-daemon-4scq8" (UID: "e328c2d0-1baf-4aba-8b4f-63c8accb0d29") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991844 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991876 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovnkube-config\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991917 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87e0c5f4-8b4b-4164-9d12-ebafec319642-hosts-file\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991950 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9209f648-9b67-4793-aeb6-250869f631b9-serviceca\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-etc-kubernetes\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.991979 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-etc-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992011 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-etc-kubernetes\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992017 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-cni-bin\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/675a1b63-8414-428b-a1ec-4084d018e04c-agent-certs\") pod \"konnectivity-agent-jm5gm\" (UID: \"675a1b63-8414-428b-a1ec-4084d018e04c\") " pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992022 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87e0c5f4-8b4b-4164-9d12-ebafec319642-hosts-file\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992056 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-etc-openvswitch\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992078 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-cni-bin\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992096 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-socket-dir-parent\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992123 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-tuned\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992149 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b80262b-6922-490c-a7ec-bf5ec7886653-tmp\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992176 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992200 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-multus-socket-dir-parent\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992200 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-systemd\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992246 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9hs\" (UniqueName: \"kubernetes.io/projected/87e0c5f4-8b4b-4164-9d12-ebafec319642-kube-api-access-9l9hs\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:45.993803 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:45.992276 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992286 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a555c7cc-d31b-47d5-9225-42a763aedc80-cni-binary-copy\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992307 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-run-systemd\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-multus-certs\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:45.992347 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret podName:2402f312-cad4-44fa-a9ae-b752b73c6eb3 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:46.492327247 +0000 UTC m=+2.962653425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret") pod "global-pull-secret-syncer-t54n4" (UID: "2402f312-cad4-44fa-a9ae-b752b73c6eb3") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992380 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a555c7cc-d31b-47d5-9225-42a763aedc80-host-run-multus-certs\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992394 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-sys\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992431 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4ld\" (UniqueName: \"kubernetes.io/projected/3b80262b-6922-490c-a7ec-bf5ec7886653-kube-api-access-fw4ld\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992442 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovnkube-config\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992465 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw9mq\" (UniqueName: \"kubernetes.io/projected/a84c227e-c638-4ada-a153-55a08282caab-kube-api-access-tw9mq\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992483 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992518 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992552 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-env-overrides\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992577 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovn-node-metrics-cert\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992619 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysctl-conf\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992624 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f22a409-745d-4c35-9662-3c0f05e2b8c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992819 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a555c7cc-d31b-47d5-9225-42a763aedc80-cni-binary-copy\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:45.994771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.992960 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f22a409-745d-4c35-9662-3c0f05e2b8c7-env-overrides\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.996037 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.996015 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f22a409-745d-4c35-9662-3c0f05e2b8c7-ovn-node-metrics-cert\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:45.996155 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:45.996138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/675a1b63-8414-428b-a1ec-4084d018e04c-agent-certs\") pod \"konnectivity-agent-jm5gm\" (UID: \"675a1b63-8414-428b-a1ec-4084d018e04c\") " pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:56:46.012849 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.012813 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54w8b\" (UniqueName: \"kubernetes.io/projected/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-kube-api-access-54w8b\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:46.012960 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.012847 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5ql\" (UniqueName: \"kubernetes.io/projected/9209f648-9b67-4793-aeb6-250869f631b9-kube-api-access-zf5ql\") pod \"node-ca-7gcp5\" (UID: \"9209f648-9b67-4793-aeb6-250869f631b9\") " pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:46.013101 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.013078 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjtb\" (UniqueName: \"kubernetes.io/projected/a555c7cc-d31b-47d5-9225-42a763aedc80-kube-api-access-txjtb\") pod \"multus-hvrq8\" (UID: \"a555c7cc-d31b-47d5-9225-42a763aedc80\") " pod="openshift-multus/multus-hvrq8" Jul 27 16:56:46.013202 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.013149 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9hs\" (UniqueName: \"kubernetes.io/projected/87e0c5f4-8b4b-4164-9d12-ebafec319642-kube-api-access-9l9hs\") pod \"node-resolver-wv5xc\" (UID: \"87e0c5f4-8b4b-4164-9d12-ebafec319642\") " pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:46.014312 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.014292 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4627z\" (UniqueName: \"kubernetes.io/projected/3f22a409-745d-4c35-9662-3c0f05e2b8c7-kube-api-access-4627z\") pod \"ovnkube-node-ntgl8\" (UID: \"3f22a409-745d-4c35-9662-3c0f05e2b8c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:46.063482 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.063432 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" event={"ID":"52b3e30e20b3d80b25b1dc1975a73979","Type":"ContainerStarted","Data":"4a1b0ad5e8d66e0057d4bb18ed7cf3046bb71382f2eec796633317f746a0015f"} Jul 27 16:56:46.064770 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.064744 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" event={"ID":"e07fcea3b53729941e121c6d1709b9c1","Type":"ContainerStarted","Data":"505720529e62cb2ef57dfda272695328dc10b58d09d740ccd4faa175a4093931"} Jul 27 16:56:46.092895 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.092868 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.093021 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.092911 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-sys-fs\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.093021 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.092933 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-cni-binary-copy\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093021 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.092960 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysctl-d\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093021 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.092990 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-run\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093021 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093015 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-registration-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093050 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-system-cni-dir\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093073 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-tuned\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093094 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b80262b-6922-490c-a7ec-bf5ec7886653-tmp\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093130 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-sys\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4ld\" (UniqueName: \"kubernetes.io/projected/3b80262b-6922-490c-a7ec-bf5ec7886653-kube-api-access-fw4ld\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093177 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw9mq\" (UniqueName: \"kubernetes.io/projected/a84c227e-c638-4ada-a153-55a08282caab-kube-api-access-tw9mq\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093201 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysctl-conf\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093231 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysctl-d\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093240 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-cnibin\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-iptables-alerter-script\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093315 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-run\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093327 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093351 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysconfig\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093370 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-kubernetes\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093385 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-lib-modules\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093391 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-system-cni-dir\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093433 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-modprobe-d\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-systemd\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093472 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn77p\" (UniqueName: \"kubernetes.io/projected/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-kube-api-access-kn77p\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093490 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-host\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093507 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093532 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-socket-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093549 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-os-release\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093623 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093660 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-kubernetes\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093663 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td96j\" (UniqueName: \"kubernetes.io/projected/d057148b-ad2c-4737-b7b0-0938b54fdd93-kube-api-access-td96j\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.093812 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093736 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-host-slash\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093781 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-var-lib-kubelet\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093371 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-registration-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093811 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-device-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093835 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093678 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-sys\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093285 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-cnibin\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.093931 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-lib-modules\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-modprobe-d\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094124 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-var-lib-kubelet\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094164 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-host\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-host-slash\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094266 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysctl-conf\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094289 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d057148b-ad2c-4737-b7b0-0938b54fdd93-os-release\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094319 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094349 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-sysconfig\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.094690 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094366 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-device-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.095421 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094436 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.095421 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094448 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-systemd\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.095421 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094450 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-sys-fs\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.095421 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094492 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a84c227e-c638-4ada-a153-55a08282caab-socket-dir\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.095421 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094539 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.095421 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094679 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-cni-binary-copy\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.095421 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.094923 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d057148b-ad2c-4737-b7b0-0938b54fdd93-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.095829 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.095631 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-iptables-alerter-script\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:46.096579 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.096555 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b80262b-6922-490c-a7ec-bf5ec7886653-etc-tuned\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.096662 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.096587 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b80262b-6922-490c-a7ec-bf5ec7886653-tmp\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.104009 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.103878 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4ld\" (UniqueName: \"kubernetes.io/projected/3b80262b-6922-490c-a7ec-bf5ec7886653-kube-api-access-fw4ld\") pod \"tuned-dpxmq\" (UID: \"3b80262b-6922-490c-a7ec-bf5ec7886653\") " pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.104114 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.104063 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:56:46.104114 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.104087 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:56:46.104114 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.104101 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8zfgj for pod openshift-network-diagnostics/network-check-target-7xqlz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:46.104305 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.104162 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj podName:acf25ffa-614a-417b-a565-e97ebcc3d76c nodeName:}" failed. No retries permitted until 2026-07-27 16:56:46.60414366 +0000 UTC m=+3.074469854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8zfgj" (UniqueName: "kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj") pod "network-check-target-7xqlz" (UID: "acf25ffa-614a-417b-a565-e97ebcc3d76c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:46.104823 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.104781 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn77p\" (UniqueName: \"kubernetes.io/projected/e721d3d5-bbcd-47eb-a1a1-73a21ddddba9-kube-api-access-kn77p\") pod \"iptables-alerter-2hfzd\" (UID: \"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9\") " pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:46.106673 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.106644 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td96j\" (UniqueName: \"kubernetes.io/projected/d057148b-ad2c-4737-b7b0-0938b54fdd93-kube-api-access-td96j\") pod \"multus-additional-cni-plugins-rlcft\" (UID: \"d057148b-ad2c-4737-b7b0-0938b54fdd93\") " pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.107030 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.107003 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw9mq\" (UniqueName: \"kubernetes.io/projected/a84c227e-c638-4ada-a153-55a08282caab-kube-api-access-tw9mq\") pod \"aws-ebs-csi-driver-node-jfblw\" (UID: \"a84c227e-c638-4ada-a153-55a08282caab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.177574 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.177520 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wv5xc" Jul 27 16:56:46.185865 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.185839 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:56:46.190564 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.190539 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e0c5f4_8b4b_4164_9d12_ebafec319642.slice/crio-e8d64328c291114519bf00a3f2b4a07dc78b7a8c72e35b3ed86975db9d9b1de6 WatchSource:0}: Error finding container e8d64328c291114519bf00a3f2b4a07dc78b7a8c72e35b3ed86975db9d9b1de6: Status 404 returned error can't find the container with id e8d64328c291114519bf00a3f2b4a07dc78b7a8c72e35b3ed86975db9d9b1de6 Jul 27 16:56:46.196010 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.195986 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f22a409_745d_4c35_9662_3c0f05e2b8c7.slice/crio-9a5e8fd085a41356c8fa46e317748b6643ff8a9c81a9c358b5c3ce2a9b0daf4c WatchSource:0}: Error finding container 9a5e8fd085a41356c8fa46e317748b6643ff8a9c81a9c358b5c3ce2a9b0daf4c: Status 404 returned error can't find the container with id 9a5e8fd085a41356c8fa46e317748b6643ff8a9c81a9c358b5c3ce2a9b0daf4c Jul 27 16:56:46.196122 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.196017 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7gcp5" Jul 27 16:56:46.202453 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.202175 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:56:46.209056 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.209019 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hvrq8" Jul 27 16:56:46.211939 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.211795 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9209f648_9b67_4793_aeb6_250869f631b9.slice/crio-c3b05eed851833e3cb8b2d5be13000ea42ce39ad555e482cde9d38294a13847b WatchSource:0}: Error finding container c3b05eed851833e3cb8b2d5be13000ea42ce39ad555e482cde9d38294a13847b: Status 404 returned error can't find the container with id c3b05eed851833e3cb8b2d5be13000ea42ce39ad555e482cde9d38294a13847b Jul 27 16:56:46.217400 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.217379 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" Jul 27 16:56:46.217992 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.217955 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675a1b63_8414_428b_a1ec_4084d018e04c.slice/crio-361eba20bef86d0520704cd22b06ec86ca932061817a16c12576c9b9cd7946e5 WatchSource:0}: Error finding container 361eba20bef86d0520704cd22b06ec86ca932061817a16c12576c9b9cd7946e5: Status 404 returned error can't find the container with id 361eba20bef86d0520704cd22b06ec86ca932061817a16c12576c9b9cd7946e5 Jul 27 16:56:46.220189 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.220166 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555c7cc_d31b_47d5_9225_42a763aedc80.slice/crio-ddd258a194acb63a23c297a8d7dacc79e6c3ae761b23e9dc32ab12b8574d2a8c WatchSource:0}: Error finding container ddd258a194acb63a23c297a8d7dacc79e6c3ae761b23e9dc32ab12b8574d2a8c: Status 404 returned error can't find the container with id ddd258a194acb63a23c297a8d7dacc79e6c3ae761b23e9dc32ab12b8574d2a8c Jul 27 16:56:46.224337 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.224325 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" Jul 27 16:56:46.227621 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.227598 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80262b_6922_490c_a7ec_bf5ec7886653.slice/crio-6cc3d70093d03f38eb394f6175128cdbb9ff0f785fb183569fbc17a553e3016b WatchSource:0}: Error finding container 6cc3d70093d03f38eb394f6175128cdbb9ff0f785fb183569fbc17a553e3016b: Status 404 returned error can't find the container with id 6cc3d70093d03f38eb394f6175128cdbb9ff0f785fb183569fbc17a553e3016b Jul 27 16:56:46.232145 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.231893 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rlcft" Jul 27 16:56:46.236982 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.236956 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84c227e_c638_4ada_a153_55a08282caab.slice/crio-e4b9afc2174af0a0361457f659f888f7d447b0a031e5b289d14b534ff5ba5587 WatchSource:0}: Error finding container e4b9afc2174af0a0361457f659f888f7d447b0a031e5b289d14b534ff5ba5587: Status 404 returned error can't find the container with id e4b9afc2174af0a0361457f659f888f7d447b0a031e5b289d14b534ff5ba5587 Jul 27 16:56:46.238187 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.238163 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2hfzd" Jul 27 16:56:46.245765 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.245740 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd057148b_ad2c_4737_b7b0_0938b54fdd93.slice/crio-2ce45e188d4e9a0ad3f581dc083998ce6b994200a95c102966a686b617edfb8d WatchSource:0}: Error finding container 2ce45e188d4e9a0ad3f581dc083998ce6b994200a95c102966a686b617edfb8d: Status 404 returned error can't find the container with id 2ce45e188d4e9a0ad3f581dc083998ce6b994200a95c102966a686b617edfb8d Jul 27 16:56:46.249127 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:56:46.249106 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode721d3d5_bbcd_47eb_a1a1_73a21ddddba9.slice/crio-f3daf36067dbfca1d76e373a62666bc77e000d6ffc9b343be5ce129948707ee2 WatchSource:0}: Error finding container f3daf36067dbfca1d76e373a62666bc77e000d6ffc9b343be5ce129948707ee2: Status 404 returned error can't find the container with id f3daf36067dbfca1d76e373a62666bc77e000d6ffc9b343be5ce129948707ee2 Jul 27 16:56:46.497373 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.497286 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:46.497373 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.497328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:46.497607 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.497443 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:46.497607 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.497454 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:46.497607 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.497503 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret podName:2402f312-cad4-44fa-a9ae-b752b73c6eb3 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:47.497487121 +0000 UTC m=+3.967813295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret") pod "global-pull-secret-syncer-t54n4" (UID: "2402f312-cad4-44fa-a9ae-b752b73c6eb3") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:46.497607 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.497526 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs podName:e328c2d0-1baf-4aba-8b4f-63c8accb0d29 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:47.497515252 +0000 UTC m=+3.967841432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs") pod "network-metrics-daemon-4scq8" (UID: "e328c2d0-1baf-4aba-8b4f-63c8accb0d29") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:46.699563 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.699457 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:46.699773 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.699747 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:56:46.699865 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.699786 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:56:46.699865 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.699805 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8zfgj for pod openshift-network-diagnostics/network-check-target-7xqlz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:46.699965 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:46.699874 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj podName:acf25ffa-614a-417b-a565-e97ebcc3d76c nodeName:}" failed. No retries permitted until 2026-07-27 16:56:47.699856072 +0000 UTC m=+4.170182270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zfgj" (UniqueName: "kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj") pod "network-check-target-7xqlz" (UID: "acf25ffa-614a-417b-a565-e97ebcc3d76c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:46.946173 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.946056 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:51:46 +0000 UTC" deadline="2028-04-22 12:06:50.15097376 +0000 UTC" Jul 27 16:56:46.946173 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:46.946119 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15235h10m3.204859059s" Jul 27 16:56:47.058296 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.058256 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:47.058418 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.058392 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:56:47.069409 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.069311 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" event={"ID":"a84c227e-c638-4ada-a153-55a08282caab","Type":"ContainerStarted","Data":"e4b9afc2174af0a0361457f659f888f7d447b0a031e5b289d14b534ff5ba5587"} Jul 27 16:56:47.071446 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.071350 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jm5gm" event={"ID":"675a1b63-8414-428b-a1ec-4084d018e04c","Type":"ContainerStarted","Data":"361eba20bef86d0520704cd22b06ec86ca932061817a16c12576c9b9cd7946e5"} Jul 27 16:56:47.074075 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.074046 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"9a5e8fd085a41356c8fa46e317748b6643ff8a9c81a9c358b5c3ce2a9b0daf4c"} Jul 27 16:56:47.083790 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.083767 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2hfzd" event={"ID":"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9","Type":"ContainerStarted","Data":"f3daf36067dbfca1d76e373a62666bc77e000d6ffc9b343be5ce129948707ee2"} Jul 27 16:56:47.094849 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.094821 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" event={"ID":"3b80262b-6922-490c-a7ec-bf5ec7886653","Type":"ContainerStarted","Data":"6cc3d70093d03f38eb394f6175128cdbb9ff0f785fb183569fbc17a553e3016b"} Jul 27 16:56:47.108560 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.108533 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hvrq8" event={"ID":"a555c7cc-d31b-47d5-9225-42a763aedc80","Type":"ContainerStarted","Data":"ddd258a194acb63a23c297a8d7dacc79e6c3ae761b23e9dc32ab12b8574d2a8c"} Jul 27 16:56:47.121460 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.121434 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7gcp5" event={"ID":"9209f648-9b67-4793-aeb6-250869f631b9","Type":"ContainerStarted","Data":"c3b05eed851833e3cb8b2d5be13000ea42ce39ad555e482cde9d38294a13847b"} Jul 27 16:56:47.131185 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.131156 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wv5xc" event={"ID":"87e0c5f4-8b4b-4164-9d12-ebafec319642","Type":"ContainerStarted","Data":"e8d64328c291114519bf00a3f2b4a07dc78b7a8c72e35b3ed86975db9d9b1de6"} Jul 27 16:56:47.136456 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.136430 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlcft" event={"ID":"d057148b-ad2c-4737-b7b0-0938b54fdd93","Type":"ContainerStarted","Data":"2ce45e188d4e9a0ad3f581dc083998ce6b994200a95c102966a686b617edfb8d"} Jul 27 16:56:47.507379 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.507340 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:47.507583 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.507408 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:47.507583 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.507524 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:47.507583 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.507571 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:47.507745 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.507610 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs podName:e328c2d0-1baf-4aba-8b4f-63c8accb0d29 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:49.507588471 +0000 UTC m=+5.977914650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs") pod "network-metrics-daemon-4scq8" (UID: "e328c2d0-1baf-4aba-8b4f-63c8accb0d29") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:47.507745 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.507636 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret podName:2402f312-cad4-44fa-a9ae-b752b73c6eb3 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:49.507625443 +0000 UTC m=+5.977951617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret") pod "global-pull-secret-syncer-t54n4" (UID: "2402f312-cad4-44fa-a9ae-b752b73c6eb3") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:47.687852 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.687817 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 16:56:47.708607 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.708455 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:47.708780 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.708672 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:56:47.708780 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.708698 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:56:47.708780 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.708713 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8zfgj for pod openshift-network-diagnostics/network-check-target-7xqlz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:47.708780 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:47.708772 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj podName:acf25ffa-614a-417b-a565-e97ebcc3d76c nodeName:}" failed. No retries permitted until 2026-07-27 16:56:49.708753684 +0000 UTC m=+6.179079865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zfgj" (UniqueName: "kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj") pod "network-check-target-7xqlz" (UID: "acf25ffa-614a-417b-a565-e97ebcc3d76c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:47.947112 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.946985 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:51:46 +0000 UTC" deadline="2028-03-03 15:52:22.702680323 +0000 UTC" Jul 27 16:56:47.947112 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:47.947060 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14038h55m34.755625465s" Jul 27 16:56:48.058296 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:48.058263 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:48.058459 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:48.058404 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:56:48.061262 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:48.058898 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:48.061262 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:48.059016 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:56:48.160008 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:48.159109 2557 generic.go:356] "Generic (PLEG): container finished" podID="52b3e30e20b3d80b25b1dc1975a73979" containerID="0ec0eccb400ae749092491e2a1369ea283194cba4c42a7663f195ddfb8376400" exitCode=0 Jul 27 16:56:48.160008 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:48.159166 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" event={"ID":"52b3e30e20b3d80b25b1dc1975a73979","Type":"ContainerDied","Data":"0ec0eccb400ae749092491e2a1369ea283194cba4c42a7663f195ddfb8376400"} Jul 27 16:56:49.058443 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:49.058408 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:49.058941 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.058562 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:56:49.525208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:49.525087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:49.525208 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:49.525156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:49.525438 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.525336 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:49.525438 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.525404 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret podName:2402f312-cad4-44fa-a9ae-b752b73c6eb3 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:53.525384189 +0000 UTC m=+9.995710365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret") pod "global-pull-secret-syncer-t54n4" (UID: "2402f312-cad4-44fa-a9ae-b752b73c6eb3") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:49.525720 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.525691 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:49.525875 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.525778 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs podName:e328c2d0-1baf-4aba-8b4f-63c8accb0d29 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:53.525750013 +0000 UTC m=+9.996076193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs") pod "network-metrics-daemon-4scq8" (UID: "e328c2d0-1baf-4aba-8b4f-63c8accb0d29") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:49.727412 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:49.727376 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:49.727642 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.727617 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:56:49.727725 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.727654 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:56:49.727725 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.727671 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8zfgj for pod openshift-network-diagnostics/network-check-target-7xqlz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:49.727836 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:49.727804 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj podName:acf25ffa-614a-417b-a565-e97ebcc3d76c nodeName:}" failed. No retries permitted until 2026-07-27 16:56:53.727782025 +0000 UTC m=+10.198108224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zfgj" (UniqueName: "kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj") pod "network-check-target-7xqlz" (UID: "acf25ffa-614a-417b-a565-e97ebcc3d76c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:50.059200 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:50.059116 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:50.059678 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:50.059283 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:56:50.059747 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:50.059705 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:50.059846 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:50.059817 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:56:51.059274 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:51.058706 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:51.059274 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:51.058846 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:56:52.062003 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:52.061965 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:52.062443 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:52.062106 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:56:52.062443 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:52.062118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:52.062443 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:52.062232 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:56:53.058374 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:53.058337 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:53.058545 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.058475 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:56:53.564714 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:53.563873 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:53.564714 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:53.563990 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:53.564714 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.564129 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:53.564714 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.564214 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs podName:e328c2d0-1baf-4aba-8b4f-63c8accb0d29 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:01.564193067 +0000 UTC m=+18.034519262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs") pod "network-metrics-daemon-4scq8" (UID: "e328c2d0-1baf-4aba-8b4f-63c8accb0d29") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:56:53.564714 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.564620 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:53.564714 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.564678 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret podName:2402f312-cad4-44fa-a9ae-b752b73c6eb3 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:01.564662868 +0000 UTC m=+18.034989044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret") pod "global-pull-secret-syncer-t54n4" (UID: "2402f312-cad4-44fa-a9ae-b752b73c6eb3") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:56:53.765457 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:53.765419 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:53.765647 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.765593 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:56:53.765647 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.765613 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:56:53.765647 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.765626 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8zfgj for pod openshift-network-diagnostics/network-check-target-7xqlz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:53.765806 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:53.765687 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj podName:acf25ffa-614a-417b-a565-e97ebcc3d76c nodeName:}" failed. No retries permitted until 2026-07-27 16:57:01.765667691 +0000 UTC m=+18.235993866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zfgj" (UniqueName: "kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj") pod "network-check-target-7xqlz" (UID: "acf25ffa-614a-417b-a565-e97ebcc3d76c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:56:54.060556 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:54.058449 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:54.060556 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:54.058607 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:56:54.060556 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:54.060296 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:54.060556 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:54.060388 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:56:55.059079 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:55.059036 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:55.059566 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:55.059191 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:56:56.058290 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:56.058253 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:56.058479 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:56.058400 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:56:56.058909 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:56.058886 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:56.059019 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:56.058994 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:56:57.058902 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:57.058868 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:57.059405 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:57.059001 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:56:58.058307 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:58.058133 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:56:58.058307 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:58.058189 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:56:58.058529 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:58.058303 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:56:58.058726 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:58.058690 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:56:59.058996 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:56:59.058932 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:56:59.059379 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:56:59.059086 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:00.058326 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:00.058289 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:00.058498 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:00.058293 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:00.058498 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:00.058431 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:00.058617 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:00.058497 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:01.059066 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:01.059032 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:01.059509 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.059143 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:01.626211 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:01.626187 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:01.626350 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:01.626228 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:01.626350 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.626320 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:57:01.626350 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.626332 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:57:01.626498 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.626378 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret podName:2402f312-cad4-44fa-a9ae-b752b73c6eb3 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:17.626359149 +0000 UTC m=+34.096685322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret") pod "global-pull-secret-syncer-t54n4" (UID: "2402f312-cad4-44fa-a9ae-b752b73c6eb3") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:57:01.626498 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.626393 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs podName:e328c2d0-1baf-4aba-8b4f-63c8accb0d29 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:17.626385327 +0000 UTC m=+34.096711499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs") pod "network-metrics-daemon-4scq8" (UID: "e328c2d0-1baf-4aba-8b4f-63c8accb0d29") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:57:01.828611 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:01.828569 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:01.828780 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.828757 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:57:01.828843 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.828789 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:57:01.828843 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.828803 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8zfgj for pod openshift-network-diagnostics/network-check-target-7xqlz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:57:01.828928 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:01.828870 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj podName:acf25ffa-614a-417b-a565-e97ebcc3d76c nodeName:}" failed. No retries permitted until 2026-07-27 16:57:17.82884877 +0000 UTC m=+34.299174943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zfgj" (UniqueName: "kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj") pod "network-check-target-7xqlz" (UID: "acf25ffa-614a-417b-a565-e97ebcc3d76c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:57:02.058835 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:02.058802 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:02.058835 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:02.058824 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:02.059067 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:02.058944 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:02.059067 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:02.059034 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:03.058228 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:03.058192 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:03.058407 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:03.058321 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:04.059577 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:04.059540 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:04.059577 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:04.059587 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:04.060115 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:04.059662 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:04.060115 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:04.059802 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:05.058855 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:05.058816 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:05.059044 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:05.058953 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:06.059156 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.058937 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:06.059732 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.058983 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:06.059732 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:06.059283 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:06.059732 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:06.059384 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:06.197992 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.197956 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"5c48a462ce468439e7972faf295dd9a9847eb3579c1d95f3fc9ce7e266874e98"} Jul 27 16:57:06.198188 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.198045 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"78dbdc227c432405e6dcc72c0339fd70e5c396c2623c3fff3b33ae0f3a4d713d"} Jul 27 16:57:06.198188 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.198095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"109c863fa31cc7b9cd7eda6e0687633fef560ce32b3141b7f37b747a071d95f6"} Jul 27 16:57:06.198188 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.198138 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"86ae5b1c28114a481baf5631c3352da57cb57ebdc31c3326dc0f6485ace14ea2"} Jul 27 16:57:06.198353 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.198195 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"0a1c51764982eed9e93b7b73ebe1e549edbc1dada6c90842f4923ee2c051a9b8"} Jul 27 16:57:06.198353 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.198243 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"a9211ea1d928cb017efa8a856db49b24acea6e0e2c2e636b5309505472e27364"} Jul 27 16:57:06.200160 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.200131 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" event={"ID":"52b3e30e20b3d80b25b1dc1975a73979","Type":"ContainerStarted","Data":"6b677fea79bf31e6984efe53c5d8dbb648c38fa3b0d41608e8a2ec2121d14dfd"} Jul 27 16:57:06.201979 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.201954 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" event={"ID":"3b80262b-6922-490c-a7ec-bf5ec7886653","Type":"ContainerStarted","Data":"d3dff7e81c762dfe6d63222626653f19f8a8ece3695c442553711b43ac0aca11"} Jul 27 16:57:06.203785 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.203757 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hvrq8" event={"ID":"a555c7cc-d31b-47d5-9225-42a763aedc80","Type":"ContainerStarted","Data":"5bdd3e9cc4019a8ea2d0483f7d19c83dd4d18c0facd26194f816cf5c030798f5"} Jul 27 16:57:06.205814 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.205790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7gcp5" event={"ID":"9209f648-9b67-4793-aeb6-250869f631b9","Type":"ContainerStarted","Data":"bb217b32bb833f2156eaebded3c63d9dbd634b3f2a1e8df3e5ca0b0fa7cb1b35"} Jul 27 16:57:06.207456 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.207427 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wv5xc" event={"ID":"87e0c5f4-8b4b-4164-9d12-ebafec319642","Type":"ContainerStarted","Data":"a0a5758b8755dfaa20c1773b3df6c9e707414d2ceac98975088f1d7e52c0c367"} Jul 27 16:57:06.209231 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.209207 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" event={"ID":"e07fcea3b53729941e121c6d1709b9c1","Type":"ContainerStarted","Data":"30dc5120cf9da251d71f2e45e40a6451475548a0f19322fbd5f2c37e13938546"} Jul 27 16:57:06.211156 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.211135 2557 generic.go:356] "Generic (PLEG): container finished" podID="d057148b-ad2c-4737-b7b0-0938b54fdd93" containerID="f8916e9574160fcaa742e8dcbafa33e97b979def5e84e856c2f85bbc18f11c99" exitCode=0 Jul 27 16:57:06.211272 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.211210 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlcft" event={"ID":"d057148b-ad2c-4737-b7b0-0938b54fdd93","Type":"ContainerDied","Data":"f8916e9574160fcaa742e8dcbafa33e97b979def5e84e856c2f85bbc18f11c99"} Jul 27 16:57:06.212879 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.212858 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" event={"ID":"a84c227e-c638-4ada-a153-55a08282caab","Type":"ContainerStarted","Data":"0bd78a89b447ad9da2223b0cc3c890b4471f5c8c9cb390e18a5c96299141e436"} Jul 27 16:57:06.214623 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.214600 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jm5gm" event={"ID":"675a1b63-8414-428b-a1ec-4084d018e04c","Type":"ContainerStarted","Data":"46700d09a8627facb7b87171fdaef6e483e118ada891fbde45bf615b3c9c7876"} Jul 27 16:57:06.247552 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.247501 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-46.ec2.internal" podStartSLOduration=21.247480471 podStartE2EDuration="21.247480471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:06.218659826 +0000 UTC m=+22.688986022" watchObservedRunningTime="2026-07-27 16:57:06.247480471 +0000 UTC m=+22.717806669" Jul 27 16:57:06.289159 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.289097 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7gcp5" podStartSLOduration=3.135495653 podStartE2EDuration="22.289073279s" totalImagesPullingTime="19.153577626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:06.289002323 +0000 UTC m=+22.759328520" watchObservedRunningTime="2026-07-27 16:57:06.289073279 +0000 UTC m=+22.759399475" Jul 27 16:57:06.289605 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.289563 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wv5xc" podStartSLOduration=3.086651946 podStartE2EDuration="22.289548137s" totalImagesPullingTime="19.202896191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:06.267548129 +0000 UTC m=+22.737874324" watchObservedRunningTime="2026-07-27 16:57:06.289548137 +0000 UTC m=+22.759874333" Jul 27 16:57:06.307062 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.307018 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jm5gm" podStartSLOduration=3.130847785 podStartE2EDuration="22.307006141s" totalImagesPullingTime="19.176158356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:06.306843159 +0000 UTC m=+22.777169352" watchObservedRunningTime="2026-07-27 16:57:06.307006141 +0000 UTC m=+22.777332347" Jul 27 16:57:06.327113 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.327065 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dpxmq" podStartSLOduration=3.188287211 podStartE2EDuration="22.327048897s" totalImagesPullingTime="19.138761686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:06.326756848 +0000 UTC m=+22.797083043" watchObservedRunningTime="2026-07-27 16:57:06.327048897 +0000 UTC m=+22.797375092" Jul 27 16:57:06.343648 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.343602 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-46.ec2.internal" podStartSLOduration=21.343585908 podStartE2EDuration="21.343585908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:06.343443402 +0000 UTC m=+22.813769608" watchObservedRunningTime="2026-07-27 16:57:06.343585908 +0000 UTC m=+22.813912104" Jul 27 16:57:06.352403 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.352356 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:57:06.353357 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.353330 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:57:06.383783 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.383726 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hvrq8" podStartSLOduration=3.141304384 podStartE2EDuration="22.38370665s" totalImagesPullingTime="19.242402266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:06.367784702 +0000 UTC m=+22.838110897" watchObservedRunningTime="2026-07-27 16:57:06.38370665 +0000 UTC m=+22.854032845" Jul 27 16:57:06.387115 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.387085 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:06.462685 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.462657 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:06.562906 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.562885 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:06.575939 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.575919 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 16:57:06.663225 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.663195 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:06.945573 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.945448 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T16:57:06.575935011Z","UUID":"379537e5-f8fc-4fbf-9cff-362e230047d4","Handler":null,"Name":"","Endpoint":""} Jul 27 16:57:06.946883 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.946866 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 16:57:06.946984 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:06.946890 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 16:57:07.058623 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.058597 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:07.058773 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:07.058714 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:07.219344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.219256 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" event={"ID":"a84c227e-c638-4ada-a153-55a08282caab","Type":"ContainerStarted","Data":"e77d997bd0346854f560dbbb2fb3f81f55ee53ead69c253a745f00b03535dfae"} Jul 27 16:57:07.220949 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.220911 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2hfzd" event={"ID":"e721d3d5-bbcd-47eb-a1a1-73a21ddddba9","Type":"ContainerStarted","Data":"4159cc342e380ab5ac67a18bfc47c5b7e9626e5bd502c9ab94ff820a07fa3f7d"} Jul 27 16:57:07.222080 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.222019 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:57:07.222211 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.222093 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jm5gm" Jul 27 16:57:07.238041 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.238001 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2hfzd" podStartSLOduration=4.083063781 podStartE2EDuration="23.237988516s" totalImagesPullingTime="19.154924735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:07.237842954 +0000 UTC m=+23.708169150" watchObservedRunningTime="2026-07-27 16:57:07.237988516 +0000 UTC m=+23.708314755" Jul 27 16:57:07.320495 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.320468 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:07.386488 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.386460 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:07.586730 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:07.586653 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:08.058231 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:08.058202 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:08.058415 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:08.058321 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:08.058415 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:08.058377 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:08.058521 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:08.058488 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:08.226666 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:08.226426 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" event={"ID":"a84c227e-c638-4ada-a153-55a08282caab","Type":"ContainerStarted","Data":"9af6da964d96a44ece8b684f0ba0d34792989e58f2c225da17e0d51058f13930"} Jul 27 16:57:08.246996 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:08.246943 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfblw" podStartSLOduration=3.475376926 podStartE2EDuration="24.246923783s" totalImagesPullingTime="20.771546857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:08.24658959 +0000 UTC m=+24.716915785" watchObservedRunningTime="2026-07-27 16:57:08.246923783 +0000 UTC m=+24.717249979" Jul 27 16:57:08.773523 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:08.773485 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:08.934734 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:08.934699 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:09.058877 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:09.058810 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:09.059027 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:09.058923 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:09.232450 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:09.232403 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"5d724565897881eed23e17385cd8658e48d9138f6103b290a5db290ea3ee70e6"} Jul 27 16:57:10.059023 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:10.058982 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:10.059200 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:10.059142 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:10.059284 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:10.059204 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:10.059347 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:10.059324 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:11.058945 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:11.058909 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:11.059498 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:11.059045 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:12.058769 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.058604 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:12.058922 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.058604 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:12.058922 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:12.058845 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:12.058992 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:12.058927 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:12.240480 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.240400 2557 generic.go:356] "Generic (PLEG): container finished" podID="d057148b-ad2c-4737-b7b0-0938b54fdd93" containerID="3de72a00daff8c655cb703d1e500cee0e4703b857a5099e57b941c30aca528fa" exitCode=0 Jul 27 16:57:12.240628 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.240486 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlcft" event={"ID":"d057148b-ad2c-4737-b7b0-0938b54fdd93","Type":"ContainerDied","Data":"3de72a00daff8c655cb703d1e500cee0e4703b857a5099e57b941c30aca528fa"} Jul 27 16:57:12.243899 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.243876 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" event={"ID":"3f22a409-745d-4c35-9662-3c0f05e2b8c7","Type":"ContainerStarted","Data":"8187767f41112d9c380338620d67ed73ce940df03b3ac9e17198d64b37c3aee4"} Jul 27 16:57:12.244135 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.244120 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:57:12.244518 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.244491 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:57:12.244621 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.244592 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:57:12.260700 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.260672 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:57:12.260818 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.260792 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:57:12.302930 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:12.302884 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" podStartSLOduration=9.014325928 podStartE2EDuration="28.302868674s" totalImagesPullingTime="19.288542746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:12.302619607 +0000 UTC m=+28.772945802" watchObservedRunningTime="2026-07-27 16:57:12.302868674 +0000 UTC m=+28.773194873" Jul 27 16:57:13.058698 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:13.058669 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:13.058930 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:13.058819 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:13.554082 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:13.554045 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:13.695515 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:13.695484 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4scq8"] Jul 27 16:57:13.695678 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:13.695612 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:13.695764 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:13.695739 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:13.698444 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:13.698418 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7xqlz"] Jul 27 16:57:13.698571 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:13.698515 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:13.698644 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:13.698622 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:13.699071 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:13.699050 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t54n4"] Jul 27 16:57:13.699192 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:13.699156 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:13.699296 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:13.699275 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:15.058634 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:15.058395 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:15.059152 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:15.058395 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:15.059152 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:15.058717 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:15.059152 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:15.058774 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:15.252302 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:15.252270 2557 generic.go:356] "Generic (PLEG): container finished" podID="d057148b-ad2c-4737-b7b0-0938b54fdd93" containerID="32164d605bb6f6d3a14524f6c16a485eb0f022cf5deaf05c3471fed7475624ac" exitCode=0 Jul 27 16:57:15.252431 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:15.252330 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlcft" event={"ID":"d057148b-ad2c-4737-b7b0-0938b54fdd93","Type":"ContainerDied","Data":"32164d605bb6f6d3a14524f6c16a485eb0f022cf5deaf05c3471fed7475624ac"} Jul 27 16:57:15.600307 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:15.600273 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:15.698105 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:15.698029 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:16.058927 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:16.058855 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:16.059408 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:16.058970 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:16.546809 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:16.546777 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:16.615337 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:16.615280 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:17.059067 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:17.059040 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:17.059544 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:17.059040 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:17.059544 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.059144 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4scq8" podUID="e328c2d0-1baf-4aba-8b4f-63c8accb0d29" Jul 27 16:57:17.059544 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.059235 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t54n4" podUID="2402f312-cad4-44fa-a9ae-b752b73c6eb3" Jul 27 16:57:17.259069 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:17.259041 2557 generic.go:356] "Generic (PLEG): container finished" podID="d057148b-ad2c-4737-b7b0-0938b54fdd93" containerID="af6a956f79d824ae7cbd60b4b215d66a7a0f868a6aed0188295b3ec4e7c29bc1" exitCode=0 Jul 27 16:57:17.259234 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:17.259095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlcft" event={"ID":"d057148b-ad2c-4737-b7b0-0938b54fdd93","Type":"ContainerDied","Data":"af6a956f79d824ae7cbd60b4b215d66a7a0f868a6aed0188295b3ec4e7c29bc1"} Jul 27 16:57:17.643446 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:17.643350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:17.643625 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:17.643480 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:17.643625 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.643422 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:57:17.643625 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.643547 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs podName:e328c2d0-1baf-4aba-8b4f-63c8accb0d29 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:49.643533394 +0000 UTC m=+66.113859582 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs") pod "network-metrics-daemon-4scq8" (UID: "e328c2d0-1baf-4aba-8b4f-63c8accb0d29") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 16:57:17.643625 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.643602 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 16:57:17.643785 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.643648 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret podName:2402f312-cad4-44fa-a9ae-b752b73c6eb3 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:49.643632576 +0000 UTC m=+66.113958766 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret") pod "global-pull-secret-syncer-t54n4" (UID: "2402f312-cad4-44fa-a9ae-b752b73c6eb3") : object "kube-system"/"original-pull-secret" not registered Jul 27 16:57:17.844714 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:17.844680 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:17.844888 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.844824 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 16:57:17.844888 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.844843 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 16:57:17.844888 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.844855 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8zfgj for pod openshift-network-diagnostics/network-check-target-7xqlz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:57:17.845054 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:17.844917 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj podName:acf25ffa-614a-417b-a565-e97ebcc3d76c nodeName:}" failed. No retries permitted until 2026-07-27 16:57:49.844899124 +0000 UTC m=+66.315225317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8zfgj" (UniqueName: "kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj") pod "network-check-target-7xqlz" (UID: "acf25ffa-614a-417b-a565-e97ebcc3d76c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 16:57:18.058474 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.058445 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:18.058611 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:18.058570 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xqlz" podUID="acf25ffa-614a-417b-a565-e97ebcc3d76c" Jul 27 16:57:18.406948 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.406924 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-46.ec2.internal" event="NodeReady" Jul 27 16:57:18.407294 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.407035 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 16:57:18.483937 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.483900 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7hlm6"] Jul 27 16:57:18.501793 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.501766 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sdkgm"] Jul 27 16:57:18.501910 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.501883 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7hlm6" Jul 27 16:57:18.504904 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.504881 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 16:57:18.505099 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.505078 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 16:57:18.505160 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.505086 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 16:57:18.506337 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.506317 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-84blb\"" Jul 27 16:57:18.513848 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.513828 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7hlm6"] Jul 27 16:57:18.513937 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.513909 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.519330 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.519308 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 16:57:18.519417 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.519361 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rkv78\"" Jul 27 16:57:18.519630 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.519618 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 16:57:18.527472 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.527442 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sdkgm"] Jul 27 16:57:18.650911 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.650873 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ac0229-6cf5-4b4d-a406-42101ea16da7-tmp-dir\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.651144 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.650925 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2789e2-beaf-4d1f-b5ff-8df0a8f53979-cert\") pod \"ingress-canary-7hlm6\" (UID: \"3b2789e2-beaf-4d1f-b5ff-8df0a8f53979\") " pod="openshift-ingress-canary/ingress-canary-7hlm6" Jul 27 16:57:18.651144 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.650943 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ac0229-6cf5-4b4d-a406-42101ea16da7-config-volume\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.651144 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.651007 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwbk\" (UniqueName: \"kubernetes.io/projected/61ac0229-6cf5-4b4d-a406-42101ea16da7-kube-api-access-grwbk\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.651144 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.651031 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4rf\" (UniqueName: \"kubernetes.io/projected/3b2789e2-beaf-4d1f-b5ff-8df0a8f53979-kube-api-access-xs4rf\") pod \"ingress-canary-7hlm6\" (UID: \"3b2789e2-beaf-4d1f-b5ff-8df0a8f53979\") " pod="openshift-ingress-canary/ingress-canary-7hlm6" Jul 27 16:57:18.651144 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.651094 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.689215 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.689124 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:18.751437 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.751411 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4rf\" (UniqueName: \"kubernetes.io/projected/3b2789e2-beaf-4d1f-b5ff-8df0a8f53979-kube-api-access-xs4rf\") pod \"ingress-canary-7hlm6\" (UID: \"3b2789e2-beaf-4d1f-b5ff-8df0a8f53979\") " pod="openshift-ingress-canary/ingress-canary-7hlm6" Jul 27 16:57:18.751591 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.751441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.751591 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.751471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ac0229-6cf5-4b4d-a406-42101ea16da7-tmp-dir\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.751591 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.751500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2789e2-beaf-4d1f-b5ff-8df0a8f53979-cert\") pod \"ingress-canary-7hlm6\" (UID: \"3b2789e2-beaf-4d1f-b5ff-8df0a8f53979\") " pod="openshift-ingress-canary/ingress-canary-7hlm6" Jul 27 16:57:18.751591 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:18.751559 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:57:18.751749 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.751607 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ac0229-6cf5-4b4d-a406-42101ea16da7-config-volume\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.751749 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:18.751630 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls podName:61ac0229-6cf5-4b4d-a406-42101ea16da7 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:19.251614383 +0000 UTC m=+35.721940569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls") pod "dns-default-sdkgm" (UID: "61ac0229-6cf5-4b4d-a406-42101ea16da7") : secret "dns-default-metrics-tls" not found Jul 27 16:57:18.751749 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.751706 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwbk\" (UniqueName: \"kubernetes.io/projected/61ac0229-6cf5-4b4d-a406-42101ea16da7-kube-api-access-grwbk\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.751882 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.751837 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ac0229-6cf5-4b4d-a406-42101ea16da7-tmp-dir\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.752114 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.752093 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ac0229-6cf5-4b4d-a406-42101ea16da7-config-volume\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.755712 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.755581 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2789e2-beaf-4d1f-b5ff-8df0a8f53979-cert\") pod \"ingress-canary-7hlm6\" (UID: \"3b2789e2-beaf-4d1f-b5ff-8df0a8f53979\") " pod="openshift-ingress-canary/ingress-canary-7hlm6" Jul 27 16:57:18.756481 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.756458 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:18.760860 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.760842 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwbk\" (UniqueName: \"kubernetes.io/projected/61ac0229-6cf5-4b4d-a406-42101ea16da7-kube-api-access-grwbk\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:18.761219 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.761167 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4rf\" (UniqueName: \"kubernetes.io/projected/3b2789e2-beaf-4d1f-b5ff-8df0a8f53979-kube-api-access-xs4rf\") pod \"ingress-canary-7hlm6\" (UID: \"3b2789e2-beaf-4d1f-b5ff-8df0a8f53979\") " pod="openshift-ingress-canary/ingress-canary-7hlm6" Jul 27 16:57:18.811241 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.811217 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7hlm6" Jul 27 16:57:18.983019 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:18.982925 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7hlm6"] Jul 27 16:57:18.987739 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:18.987713 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2789e2_beaf_4d1f_b5ff_8df0a8f53979.slice/crio-4f7866f07d4ead23649d06f2ca6f215eb2a514eecc4e1f31e20522a2856e211d WatchSource:0}: Error finding container 4f7866f07d4ead23649d06f2ca6f215eb2a514eecc4e1f31e20522a2856e211d: Status 404 returned error can't find the container with id 4f7866f07d4ead23649d06f2ca6f215eb2a514eecc4e1f31e20522a2856e211d Jul 27 16:57:19.058724 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:19.058699 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:19.058724 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:19.058716 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:19.061910 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:19.061888 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 16:57:19.062010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:19.061947 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7rsp\"" Jul 27 16:57:19.062069 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:19.061898 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 16:57:19.256538 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:19.256457 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:19.256771 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:19.256634 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:57:19.256771 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:19.256713 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls podName:61ac0229-6cf5-4b4d-a406-42101ea16da7 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:20.256689565 +0000 UTC m=+36.727015742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls") pod "dns-default-sdkgm" (UID: "61ac0229-6cf5-4b4d-a406-42101ea16da7") : secret "dns-default-metrics-tls" not found Jul 27 16:57:19.264315 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:19.264290 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7hlm6" event={"ID":"3b2789e2-beaf-4d1f-b5ff-8df0a8f53979","Type":"ContainerStarted","Data":"4f7866f07d4ead23649d06f2ca6f215eb2a514eecc4e1f31e20522a2856e211d"} Jul 27 16:57:20.058809 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:20.058766 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:20.061340 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:20.061316 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 16:57:20.061751 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:20.061728 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 16:57:20.061845 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:20.061736 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5cwf9\"" Jul 27 16:57:20.265512 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:20.265470 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:20.265671 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:20.265641 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:57:20.265779 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:20.265765 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls podName:61ac0229-6cf5-4b4d-a406-42101ea16da7 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:22.265740608 +0000 UTC m=+38.736066787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls") pod "dns-default-sdkgm" (UID: "61ac0229-6cf5-4b4d-a406-42101ea16da7") : secret "dns-default-metrics-tls" not found Jul 27 16:57:21.044513 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:21.044477 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:21.145904 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:21.145869 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:21.561469 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:21.561439 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:21.662292 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:21.662263 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:22.283339 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.283304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:22.283744 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:22.283501 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:57:22.283744 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:22.283584 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls podName:61ac0229-6cf5-4b4d-a406-42101ea16da7 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:26.283559938 +0000 UTC m=+42.753886112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls") pod "dns-default-sdkgm" (UID: "61ac0229-6cf5-4b4d-a406-42101ea16da7") : secret "dns-default-metrics-tls" not found Jul 27 16:57:22.623861 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.623780 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lr9b8"] Jul 27 16:57:22.637014 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.636989 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.639723 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.639484 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 16:57:22.639723 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.639526 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bzm7g\"" Jul 27 16:57:22.640339 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.639737 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 16:57:22.640339 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.639774 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 16:57:22.640339 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.640069 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 16:57:22.640847 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.640824 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lr9b8"] Jul 27 16:57:22.646844 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.646819 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:22.710621 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.710583 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:22.787491 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.787456 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c12e97b8-48e3-4341-a8b5-700608955f33-crio-socket\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.787491 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.787498 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.787741 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.787529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.787741 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.787646 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zwj4\" (UniqueName: \"kubernetes.io/projected/c12e97b8-48e3-4341-a8b5-700608955f33-kube-api-access-5zwj4\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.787741 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.787686 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c12e97b8-48e3-4341-a8b5-700608955f33-data-volume\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.888671 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.888627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c12e97b8-48e3-4341-a8b5-700608955f33-data-volume\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.888837 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.888751 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c12e97b8-48e3-4341-a8b5-700608955f33-crio-socket\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.888837 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.888778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.888837 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.888794 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.888837 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.888835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zwj4\" (UniqueName: \"kubernetes.io/projected/c12e97b8-48e3-4341-a8b5-700608955f33-kube-api-access-5zwj4\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.889046 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:22.888917 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:57:22.889046 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:22.888962 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:57:22.889046 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.889007 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c12e97b8-48e3-4341-a8b5-700608955f33-crio-socket\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.889046 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:22.889024 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:23.389001013 +0000 UTC m=+39.859327209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : configmap "kube-rbac-proxy" not found Jul 27 16:57:22.889046 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:22.889047 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:23.389037089 +0000 UTC m=+39.859363261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : secret "insights-runtime-extractor-tls" not found Jul 27 16:57:22.889315 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.889252 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c12e97b8-48e3-4341-a8b5-700608955f33-data-volume\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.899517 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.899392 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zwj4\" (UniqueName: \"kubernetes.io/projected/c12e97b8-48e3-4341-a8b5-700608955f33-kube-api-access-5zwj4\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:22.964877 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.964841 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:22.985033 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:22.985001 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97"] Jul 27 16:57:23.006228 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.006199 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97"] Jul 27 16:57:23.006370 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.006270 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" Jul 27 16:57:23.008803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.008579 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 27 16:57:23.008803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.008712 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hzfp8\"" Jul 27 16:57:23.009001 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.008804 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 27 16:57:23.026565 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.026530 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:23.090126 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.090088 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:23.190464 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.190404 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q77\" (UniqueName: \"kubernetes.io/projected/cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a-kube-api-access-h6q77\") pod \"migrator-6865c9967-zsd97\" (UID: \"cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" Jul 27 16:57:23.291517 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.291482 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q77\" (UniqueName: \"kubernetes.io/projected/cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a-kube-api-access-h6q77\") pod \"migrator-6865c9967-zsd97\" (UID: \"cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" Jul 27 16:57:23.300369 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.300352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q77\" (UniqueName: \"kubernetes.io/projected/cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a-kube-api-access-h6q77\") pod \"migrator-6865c9967-zsd97\" (UID: \"cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" Jul 27 16:57:23.318402 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.318376 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" Jul 27 16:57:23.361931 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.361890 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:23.392220 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.392195 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:23.392319 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.392222 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:23.392361 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:23.392326 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:57:23.392426 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:23.392411 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:57:23.392467 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:23.392414 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:24.392393927 +0000 UTC m=+40.862720125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : configmap "kube-rbac-proxy" not found Jul 27 16:57:23.392514 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:23.392487 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:24.392469956 +0000 UTC m=+40.862796136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : secret "insights-runtime-extractor-tls" not found Jul 27 16:57:23.464254 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.464216 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:23.579362 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.579314 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97"] Jul 27 16:57:23.597631 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:23.597606 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1d195d_eb03_4dc2_a1a5_5c6f2cdd925a.slice/crio-933b8838bc549240aa4b8ab0727f71a57913d2d345c4ae9bb35a63a89cd067cc WatchSource:0}: Error finding container 933b8838bc549240aa4b8ab0727f71a57913d2d345c4ae9bb35a63a89cd067cc: Status 404 returned error can't find the container with id 933b8838bc549240aa4b8ab0727f71a57913d2d345c4ae9bb35a63a89cd067cc Jul 27 16:57:23.674402 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.674363 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:23.783062 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.782989 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:23.844903 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:23.844876 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:24.068767 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.068691 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:24.159858 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.159825 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:24.279103 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.279069 2557 generic.go:356] "Generic (PLEG): container finished" podID="d057148b-ad2c-4737-b7b0-0938b54fdd93" containerID="9b2b7b8898399d22d8d6712c7592d1bd627293f96a6939c34ebf566a8d0773b2" exitCode=0 Jul 27 16:57:24.279271 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.279150 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlcft" event={"ID":"d057148b-ad2c-4737-b7b0-0938b54fdd93","Type":"ContainerDied","Data":"9b2b7b8898399d22d8d6712c7592d1bd627293f96a6939c34ebf566a8d0773b2"} Jul 27 16:57:24.280654 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.280629 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7hlm6" event={"ID":"3b2789e2-beaf-4d1f-b5ff-8df0a8f53979","Type":"ContainerStarted","Data":"d27ba74371f371a8a33fd03bc49f51862bf1e64cca6262ca360327eb65a4ae0f"} Jul 27 16:57:24.281728 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.281703 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" event={"ID":"cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a","Type":"ContainerStarted","Data":"933b8838bc549240aa4b8ab0727f71a57913d2d345c4ae9bb35a63a89cd067cc"} Jul 27 16:57:24.321311 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.321200 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7hlm6" podStartSLOduration=1.876817261 podStartE2EDuration="6.321166541s" totalImagesPullingTime="4.44434928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:24.320069921 +0000 UTC m=+40.790396115" watchObservedRunningTime="2026-07-27 16:57:24.321166541 +0000 UTC m=+40.791492752" Jul 27 16:57:24.400237 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.400204 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:24.400375 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.400252 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:24.400375 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:24.400334 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:57:24.400493 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:24.400427 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:26.400407216 +0000 UTC m=+42.870733410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : configmap "kube-rbac-proxy" not found Jul 27 16:57:24.400493 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:24.400430 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:57:24.400493 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:24.400493 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:26.400474666 +0000 UTC m=+42.870800862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : secret "insights-runtime-extractor-tls" not found Jul 27 16:57:24.777753 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:24.777726 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wv5xc_87e0c5f4-8b4b-4164-9d12-ebafec319642/dns-node-resolver/0.log" Jul 27 16:57:25.286751 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.286718 2557 generic.go:356] "Generic (PLEG): container finished" podID="d057148b-ad2c-4737-b7b0-0938b54fdd93" containerID="709bd772d901eaeb2938ab53c32deb49713df1782c8d219e8f80d4f0f31f0d90" exitCode=0 Jul 27 16:57:25.286891 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.286805 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlcft" event={"ID":"d057148b-ad2c-4737-b7b0-0938b54fdd93","Type":"ContainerDied","Data":"709bd772d901eaeb2938ab53c32deb49713df1782c8d219e8f80d4f0f31f0d90"} Jul 27 16:57:25.376780 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.376753 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7gcp5_9209f648-9b67-4793-aeb6-250869f631b9/node-ca/0.log" Jul 27 16:57:25.659699 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.659674 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-l7gkk"] Jul 27 16:57:25.683888 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.683864 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-l7gkk"] Jul 27 16:57:25.684123 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.683953 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.686430 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.686407 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 16:57:25.686430 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.686427 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 16:57:25.686592 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.686479 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n5lvq\"" Jul 27 16:57:25.686592 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.686489 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 16:57:25.686592 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.686513 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 16:57:25.688193 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.688154 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:25.745751 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.745721 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:25.811980 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.811948 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-signing-cabundle\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.812111 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.812015 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-signing-key\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.812150 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.812133 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9gx\" (UniqueName: \"kubernetes.io/projected/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-kube-api-access-pv9gx\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.913481 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.913448 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9gx\" (UniqueName: \"kubernetes.io/projected/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-kube-api-access-pv9gx\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.913642 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.913497 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-signing-cabundle\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.913713 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.913661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-signing-key\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.914161 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.914136 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-signing-cabundle\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.915981 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.915964 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-signing-key\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.922266 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.922237 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9gx\" (UniqueName: \"kubernetes.io/projected/7e5ac19b-8578-4bf6-b381-fabcd0dcead9-kube-api-access-pv9gx\") pod \"service-ca-658d7b7c78-l7gkk\" (UID: \"7e5ac19b-8578-4bf6-b381-fabcd0dcead9\") " pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:25.993005 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:25.992988 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" Jul 27 16:57:26.134530 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.134505 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-l7gkk"] Jul 27 16:57:26.137264 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:26.137243 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5ac19b_8578_4bf6_b381_fabcd0dcead9.slice/crio-2dcac1d31bba4b8f680dc5911cedbaa8e632db5781e539f16b057567994af37b WatchSource:0}: Error finding container 2dcac1d31bba4b8f680dc5911cedbaa8e632db5781e539f16b057567994af37b: Status 404 returned error can't find the container with id 2dcac1d31bba4b8f680dc5911cedbaa8e632db5781e539f16b057567994af37b Jul 27 16:57:26.177773 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.177753 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7hlm6_3b2789e2-beaf-4d1f-b5ff-8df0a8f53979/serve-healthcheck-canary/0.log" Jul 27 16:57:26.292235 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.292196 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlcft" event={"ID":"d057148b-ad2c-4737-b7b0-0938b54fdd93","Type":"ContainerStarted","Data":"bc4e0556fc1adaf791856bdd44c7bfadc7fd8d6492ae19685ab1113364c61c4c"} Jul 27 16:57:26.293369 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.293346 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" event={"ID":"7e5ac19b-8578-4bf6-b381-fabcd0dcead9","Type":"ContainerStarted","Data":"2dcac1d31bba4b8f680dc5911cedbaa8e632db5781e539f16b057567994af37b"} Jul 27 16:57:26.294830 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.294810 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" event={"ID":"cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a","Type":"ContainerStarted","Data":"9a56e37171f722324acdd61681ee85b8b0df65d71396c92171775602ccee2ad9"} Jul 27 16:57:26.294924 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.294853 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" event={"ID":"cd1d195d-eb03-4dc2-a1a5-5c6f2cdd925a","Type":"ContainerStarted","Data":"c0b13b4d9d035f0b49e91ab4880eccc770ffc2684994dbc84fe61bdf2663819e"} Jul 27 16:57:26.316885 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.316843 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rlcft" podStartSLOduration=8.1254025 podStartE2EDuration="42.316831547s" totalImagesPullingTime="34.191429047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:26.31594165 +0000 UTC m=+42.786267844" watchObservedRunningTime="2026-07-27 16:57:26.316831547 +0000 UTC m=+42.787157741" Jul 27 16:57:26.318073 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.318051 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:26.321373 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:26.318543 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:57:26.321373 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:26.318693 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls podName:61ac0229-6cf5-4b4d-a406-42101ea16da7 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:34.318640973 +0000 UTC m=+50.788967146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls") pod "dns-default-sdkgm" (UID: "61ac0229-6cf5-4b4d-a406-42101ea16da7") : secret "dns-default-metrics-tls" not found Jul 27 16:57:26.340010 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.339968 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zsd97" podStartSLOduration=1.971940273 podStartE2EDuration="4.339954815s" totalImagesPullingTime="2.368014542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:26.338847816 +0000 UTC m=+42.809174011" watchObservedRunningTime="2026-07-27 16:57:26.339954815 +0000 UTC m=+42.810281010" Jul 27 16:57:26.419521 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.419453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:26.419521 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.419486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:26.419960 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:26.419588 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:57:26.419960 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:26.419649 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:57:26.419960 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:26.419685 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:30.419663991 +0000 UTC m=+46.889990182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : configmap "kube-rbac-proxy" not found Jul 27 16:57:26.419960 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:26.419706 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:30.419697008 +0000 UTC m=+46.890023180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : secret "insights-runtime-extractor-tls" not found Jul 27 16:57:26.461016 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.460973 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:26.550154 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.550117 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:26.650939 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.650908 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:26.916817 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:26.916763 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:30.276864 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:30.276831 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:30.449856 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:30.449823 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:30.450009 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:30.449862 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:30.450009 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:30.449941 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 16:57:30.450009 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:30.449986 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:57:30.450009 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:30.450007 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:38.449992152 +0000 UTC m=+54.920318328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : configmap "kube-rbac-proxy" not found Jul 27 16:57:30.450141 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:30.450031 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:38.450019926 +0000 UTC m=+54.920346099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : secret "insights-runtime-extractor-tls" not found Jul 27 16:57:31.308188 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:31.308139 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" event={"ID":"7e5ac19b-8578-4bf6-b381-fabcd0dcead9","Type":"ContainerStarted","Data":"eb568753b5f673edc9900bd01e370f8317f8066459544a179840a796d97001b4"} Jul 27 16:57:32.119992 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:32.119940 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:32.263085 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:32.263047 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:32.363069 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:32.363036 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:32.464900 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:32.464865 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:33.024274 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:33.024243 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:33.417212 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:33.417160 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:34.014143 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:34.014106 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:34.378712 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:34.378683 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:34.378866 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:34.378834 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 16:57:34.378925 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:34.378913 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls podName:61ac0229-6cf5-4b4d-a406-42101ea16da7 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:50.37889821 +0000 UTC m=+66.849224383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls") pod "dns-default-sdkgm" (UID: "61ac0229-6cf5-4b4d-a406-42101ea16da7") : secret "dns-default-metrics-tls" not found Jul 27 16:57:34.413626 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:34.413594 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:34.815921 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:34.815834 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:34.874310 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:34.874279 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:35.018936 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:35.018903 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:36.044702 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:36.044668 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:36.102029 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:36.101999 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:36.215073 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:36.215043 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:37.092154 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:37.092118 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:37.192472 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:37.192439 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:37.261582 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:37.261553 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:37.905645 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:37.905612 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:38.217088 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:38.216987 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:38.422003 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:38.421970 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:38.509281 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:38.509202 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:38.509281 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:38.509240 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:38.509451 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:38.509404 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 16:57:38.509493 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:38.509471 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls podName:c12e97b8-48e3-4341-a8b5-700608955f33 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:54.509456638 +0000 UTC m=+70.979782812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lr9b8" (UID: "c12e97b8-48e3-4341-a8b5-700608955f33") : secret "insights-runtime-extractor-tls" not found Jul 27 16:57:38.509721 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:38.509704 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c12e97b8-48e3-4341-a8b5-700608955f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:38.613988 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:38.613958 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:39.021483 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:39.021449 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:39.423694 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:39.423661 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:39.617221 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:39.617171 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:40.212064 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:40.212009 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:40.421294 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:40.421255 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:40.814235 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:40.814168 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:41.022284 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:41.022252 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:41.619444 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:41.619406 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:42.220408 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:42.220371 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:42.415306 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:42.415275 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:42.625758 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:42.625686 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:42.818541 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:42.818487 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:44.258204 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:44.258163 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgl8" Jul 27 16:57:44.288771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:44.288723 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-l7gkk" podStartSLOduration=14.891518159 podStartE2EDuration="19.288708008s" totalImagesPullingTime="4.397189849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:31.328370147 +0000 UTC m=+47.798696343" watchObservedRunningTime="2026-07-27 16:57:44.288708008 +0000 UTC m=+60.759034204" Jul 27 16:57:49.693700 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.693648 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:49.694097 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.693755 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:49.696583 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.696562 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 16:57:49.696637 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.696584 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 16:57:49.706484 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.706453 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e328c2d0-1baf-4aba-8b4f-63c8accb0d29-metrics-certs\") pod \"network-metrics-daemon-4scq8\" (UID: \"e328c2d0-1baf-4aba-8b4f-63c8accb0d29\") " pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:49.706622 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.706457 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2402f312-cad4-44fa-a9ae-b752b73c6eb3-original-pull-secret\") pod \"global-pull-secret-syncer-t54n4\" (UID: \"2402f312-cad4-44fa-a9ae-b752b73c6eb3\") " pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:49.894852 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.894814 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:49.898284 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.898257 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj"] Jul 27 16:57:49.899897 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.899878 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 16:57:49.903224 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.903209 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9"] Jul 27 16:57:49.903363 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.903346 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" Jul 27 16:57:49.905444 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.905424 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:49.908024 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908003 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 16:57:49.908117 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908053 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 16:57:49.908117 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908070 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2nk48\"" Jul 27 16:57:49.908249 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908129 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 16:57:49.908249 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908140 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 16:57:49.908883 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908602 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 27 16:57:49.908883 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908692 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 27 16:57:49.908883 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908717 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 27 16:57:49.908883 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908785 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 27 16:57:49.908883 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.908847 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 27 16:57:49.915413 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.915394 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj"] Jul 27 16:57:49.918222 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.918203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfgj\" (UniqueName: \"kubernetes.io/projected/acf25ffa-614a-417b-a565-e97ebcc3d76c-kube-api-access-8zfgj\") pod \"network-check-target-7xqlz\" (UID: \"acf25ffa-614a-417b-a565-e97ebcc3d76c\") " pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:49.921349 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.921325 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9"] Jul 27 16:57:49.972822 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.972757 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t54n4" Jul 27 16:57:49.973832 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.973801 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:49.980706 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.980691 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7rsp\"" Jul 27 16:57:49.989607 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.989585 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4scq8" Jul 27 16:57:49.994324 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.994302 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d85f5b8c7-b6vcz"] Jul 27 16:57:49.997046 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.996977 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-hub\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:49.997156 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.997066 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94r7h\" (UniqueName: \"kubernetes.io/projected/f4626394-7b10-4288-82f6-a61e9a170203-kube-api-access-94r7h\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:49.997156 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.997119 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:49.997282 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.997206 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/025239ef-c6af-4ec7-9eba-458e07a845d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj\" (UID: \"025239ef-c6af-4ec7-9eba-458e07a845d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" Jul 27 16:57:49.997282 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.997232 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55f84\" (UniqueName: \"kubernetes.io/projected/025239ef-c6af-4ec7-9eba-458e07a845d8-kube-api-access-55f84\") pod \"managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj\" (UID: \"025239ef-c6af-4ec7-9eba-458e07a845d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" Jul 27 16:57:49.997282 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.997266 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:49.997427 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.997356 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-ca\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:49.997427 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.997397 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4626394-7b10-4288-82f6-a61e9a170203-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:49.997888 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:49.997755 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.001274 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.001254 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh2hl\"" Jul 27 16:57:50.001274 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.001265 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 16:57:50.001501 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.001274 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 16:57:50.001501 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.001256 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 16:57:50.007783 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.007757 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 16:57:50.012595 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.012538 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d85f5b8c7-b6vcz"] Jul 27 16:57:50.044340 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.044279 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d85f5b8c7-b6vcz"] Jul 27 16:57:50.044597 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:50.044538 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-68wxv registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-68wxv registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" podUID="160e821a-575f-4aa6-8552-0b333f613c77" Jul 27 16:57:50.073151 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.073132 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5cwf9\"" Jul 27 16:57:50.081422 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.081398 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:50.096386 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.096361 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:50.098490 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098462 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-image-registry-private-configuration\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.098561 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098506 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/160e821a-575f-4aa6-8552-0b333f613c77-ca-trust-extracted\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.098608 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098564 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-registry-tls\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.098608 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-hub\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.098696 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098621 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94r7h\" (UniqueName: \"kubernetes.io/projected/f4626394-7b10-4288-82f6-a61e9a170203-kube-api-access-94r7h\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.098696 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098644 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-trusted-ca\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.098696 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098679 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.098828 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098762 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/025239ef-c6af-4ec7-9eba-458e07a845d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj\" (UID: \"025239ef-c6af-4ec7-9eba-458e07a845d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" Jul 27 16:57:50.098899 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098879 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-registry-certificates\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.098958 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098939 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-ca\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.099011 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.098967 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4626394-7b10-4288-82f6-a61e9a170203-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.099916 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.099074 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-bound-sa-token\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.099916 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.099144 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-installation-pull-secrets\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.099916 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.099193 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wxv\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-kube-api-access-68wxv\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.099916 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.099264 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55f84\" (UniqueName: \"kubernetes.io/projected/025239ef-c6af-4ec7-9eba-458e07a845d8-kube-api-access-55f84\") pod \"managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj\" (UID: \"025239ef-c6af-4ec7-9eba-458e07a845d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" Jul 27 16:57:50.099916 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.099292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.100332 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.099922 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4626394-7b10-4288-82f6-a61e9a170203-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.101959 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.101933 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/025239ef-c6af-4ec7-9eba-458e07a845d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj\" (UID: \"025239ef-c6af-4ec7-9eba-458e07a845d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" Jul 27 16:57:50.102099 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.102084 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.102197 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.102100 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-ca\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.102348 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.102326 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-hub\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.102386 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.102369 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4626394-7b10-4288-82f6-a61e9a170203-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.107418 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.107394 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94r7h\" (UniqueName: \"kubernetes.io/projected/f4626394-7b10-4288-82f6-a61e9a170203-kube-api-access-94r7h\") pod \"cluster-proxy-proxy-agent-75f4f79c8b-tl6t9\" (UID: \"f4626394-7b10-4288-82f6-a61e9a170203\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.121277 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.121255 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55f84\" (UniqueName: \"kubernetes.io/projected/025239ef-c6af-4ec7-9eba-458e07a845d8-kube-api-access-55f84\") pod \"managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj\" (UID: \"025239ef-c6af-4ec7-9eba-458e07a845d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" Jul 27 16:57:50.122608 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.122582 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t54n4"] Jul 27 16:57:50.142639 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.142614 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4scq8"] Jul 27 16:57:50.145995 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:50.145968 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode328c2d0_1baf_4aba_8b4f_63c8accb0d29.slice/crio-fcb926eda3ada56733d5b8ac60855374ea1cb15dd87786b322ac28d7dff48329 WatchSource:0}: Error finding container fcb926eda3ada56733d5b8ac60855374ea1cb15dd87786b322ac28d7dff48329: Status 404 returned error can't find the container with id fcb926eda3ada56733d5b8ac60855374ea1cb15dd87786b322ac28d7dff48329 Jul 27 16:57:50.199814 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.199780 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-image-registry-private-configuration\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.199937 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.199819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/160e821a-575f-4aa6-8552-0b333f613c77-ca-trust-extracted\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.199937 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.199867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-registry-tls\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.199937 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.199892 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-trusted-ca\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.199937 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.199929 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-registry-certificates\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.200112 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.199972 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-bound-sa-token\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.200112 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.200017 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-installation-pull-secrets\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.200112 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.200047 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wxv\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-kube-api-access-68wxv\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.200530 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.200415 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/160e821a-575f-4aa6-8552-0b333f613c77-ca-trust-extracted\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.200943 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.200921 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-registry-certificates\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.201461 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.201439 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-trusted-ca\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.202323 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.202304 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-image-registry-private-configuration\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.202803 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.202789 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-registry-tls\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.202860 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.202841 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-installation-pull-secrets\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.208719 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.208698 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-bound-sa-token\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.208978 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.208960 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wxv\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-kube-api-access-68wxv\") pod \"image-registry-6d85f5b8c7-b6vcz\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.221546 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.221529 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" Jul 27 16:57:50.223310 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.223263 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7xqlz"] Jul 27 16:57:50.225998 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:50.225979 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf25ffa_614a_417b_a565_e97ebcc3d76c.slice/crio-4bf8cc1009454340dfdae6cd17b3576045fa145bbaa2440002a99997e228c203 WatchSource:0}: Error finding container 4bf8cc1009454340dfdae6cd17b3576045fa145bbaa2440002a99997e228c203: Status 404 returned error can't find the container with id 4bf8cc1009454340dfdae6cd17b3576045fa145bbaa2440002a99997e228c203 Jul 27 16:57:50.231638 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.231621 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" Jul 27 16:57:50.355584 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.355554 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7xqlz" event={"ID":"acf25ffa-614a-417b-a565-e97ebcc3d76c","Type":"ContainerStarted","Data":"4bf8cc1009454340dfdae6cd17b3576045fa145bbaa2440002a99997e228c203"} Jul 27 16:57:50.356704 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.356676 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4scq8" event={"ID":"e328c2d0-1baf-4aba-8b4f-63c8accb0d29","Type":"ContainerStarted","Data":"fcb926eda3ada56733d5b8ac60855374ea1cb15dd87786b322ac28d7dff48329"} Jul 27 16:57:50.358146 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.358125 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.358278 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.358158 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t54n4" event={"ID":"2402f312-cad4-44fa-a9ae-b752b73c6eb3","Type":"ContainerStarted","Data":"b5dd253bf4c8f715a55f1c8e1508f9c365966631fd40f70092c09e6d7a12d64b"} Jul 27 16:57:50.358914 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.358889 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj"] Jul 27 16:57:50.369912 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:50.369887 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod025239ef_c6af_4ec7_9eba_458e07a845d8.slice/crio-dc0be79602ee56842628f8b0cb914f22a8148cb1e8c95cf62d8a1888af7c6298 WatchSource:0}: Error finding container dc0be79602ee56842628f8b0cb914f22a8148cb1e8c95cf62d8a1888af7c6298: Status 404 returned error can't find the container with id dc0be79602ee56842628f8b0cb914f22a8148cb1e8c95cf62d8a1888af7c6298 Jul 27 16:57:50.371950 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.371931 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:50.374050 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.373987 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9"] Jul 27 16:57:50.377201 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:50.377160 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4626394_7b10_4288_82f6_a61e9a170203.slice/crio-1a1ea01c4db94a57b9ed0d118879490c6f5e387ed1b2c7240cbb2d35646d7f29 WatchSource:0}: Error finding container 1a1ea01c4db94a57b9ed0d118879490c6f5e387ed1b2c7240cbb2d35646d7f29: Status 404 returned error can't find the container with id 1a1ea01c4db94a57b9ed0d118879490c6f5e387ed1b2c7240cbb2d35646d7f29 Jul 27 16:57:50.401890 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.401849 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:50.403975 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.403954 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ac0229-6cf5-4b4d-a406-42101ea16da7-metrics-tls\") pod \"dns-default-sdkgm\" (UID: \"61ac0229-6cf5-4b4d-a406-42101ea16da7\") " pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:50.502709 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.502638 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-installation-pull-secrets\") pod \"160e821a-575f-4aa6-8552-0b333f613c77\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " Jul 27 16:57:50.502709 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.502685 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-image-registry-private-configuration\") pod \"160e821a-575f-4aa6-8552-0b333f613c77\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " Jul 27 16:57:50.502709 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.502705 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68wxv\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-kube-api-access-68wxv\") pod \"160e821a-575f-4aa6-8552-0b333f613c77\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " Jul 27 16:57:50.502941 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.502729 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-trusted-ca\") pod \"160e821a-575f-4aa6-8552-0b333f613c77\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " Jul 27 16:57:50.502941 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.502752 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-registry-tls\") pod \"160e821a-575f-4aa6-8552-0b333f613c77\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " Jul 27 16:57:50.502941 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.502790 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-registry-certificates\") pod \"160e821a-575f-4aa6-8552-0b333f613c77\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " Jul 27 16:57:50.502941 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.502813 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/160e821a-575f-4aa6-8552-0b333f613c77-ca-trust-extracted\") pod \"160e821a-575f-4aa6-8552-0b333f613c77\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " Jul 27 16:57:50.502941 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.502837 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-bound-sa-token\") pod \"160e821a-575f-4aa6-8552-0b333f613c77\" (UID: \"160e821a-575f-4aa6-8552-0b333f613c77\") " Jul 27 16:57:50.503333 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.503273 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160e821a-575f-4aa6-8552-0b333f613c77-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "160e821a-575f-4aa6-8552-0b333f613c77" (UID: "160e821a-575f-4aa6-8552-0b333f613c77"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:57:50.503465 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.503328 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "160e821a-575f-4aa6-8552-0b333f613c77" (UID: "160e821a-575f-4aa6-8552-0b333f613c77"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:57:50.503536 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.503481 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "160e821a-575f-4aa6-8552-0b333f613c77" (UID: "160e821a-575f-4aa6-8552-0b333f613c77"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:57:50.505016 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.504974 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "160e821a-575f-4aa6-8552-0b333f613c77" (UID: "160e821a-575f-4aa6-8552-0b333f613c77"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:57:50.505103 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.505028 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "160e821a-575f-4aa6-8552-0b333f613c77" (UID: "160e821a-575f-4aa6-8552-0b333f613c77"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:57:50.505103 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.505040 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "160e821a-575f-4aa6-8552-0b333f613c77" (UID: "160e821a-575f-4aa6-8552-0b333f613c77"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:57:50.505259 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.505241 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "160e821a-575f-4aa6-8552-0b333f613c77" (UID: "160e821a-575f-4aa6-8552-0b333f613c77"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 16:57:50.505295 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.505281 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-kube-api-access-68wxv" (OuterVolumeSpecName: "kube-api-access-68wxv") pod "160e821a-575f-4aa6-8552-0b333f613c77" (UID: "160e821a-575f-4aa6-8552-0b333f613c77"). InnerVolumeSpecName "kube-api-access-68wxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:57:50.603851 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.603821 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-registry-tls\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 16:57:50.603851 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.603846 2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-registry-certificates\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 16:57:50.603851 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.603856 2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/160e821a-575f-4aa6-8552-0b333f613c77-ca-trust-extracted\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 16:57:50.604037 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.603864 2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-bound-sa-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 16:57:50.604037 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.603874 2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-installation-pull-secrets\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 16:57:50.604037 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.603883 2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/160e821a-575f-4aa6-8552-0b333f613c77-image-registry-private-configuration\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 16:57:50.604037 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.603893 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wxv\" (UniqueName: \"kubernetes.io/projected/160e821a-575f-4aa6-8552-0b333f613c77-kube-api-access-68wxv\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 16:57:50.604037 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.603901 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/160e821a-575f-4aa6-8552-0b333f613c77-trusted-ca\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 16:57:50.626480 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.626458 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rkv78\"" Jul 27 16:57:50.632454 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.632437 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:50.763527 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.763460 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sdkgm"] Jul 27 16:57:50.766960 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:50.766928 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ac0229_6cf5_4b4d_a406_42101ea16da7.slice/crio-c5a26b7e6d4f70f5d2358a1a0177ee935ee366df4cacaeb33bbca6e026fa8c23 WatchSource:0}: Error finding container c5a26b7e6d4f70f5d2358a1a0177ee935ee366df4cacaeb33bbca6e026fa8c23: Status 404 returned error can't find the container with id c5a26b7e6d4f70f5d2358a1a0177ee935ee366df4cacaeb33bbca6e026fa8c23 Jul 27 16:57:50.862685 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.862653 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:50.969294 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:50.969256 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:51.045759 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:51.045662 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:51.369345 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:51.369259 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" event={"ID":"025239ef-c6af-4ec7-9eba-458e07a845d8","Type":"ContainerStarted","Data":"dc0be79602ee56842628f8b0cb914f22a8148cb1e8c95cf62d8a1888af7c6298"} Jul 27 16:57:51.372750 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:51.372654 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdkgm" event={"ID":"61ac0229-6cf5-4b4d-a406-42101ea16da7","Type":"ContainerStarted","Data":"c5a26b7e6d4f70f5d2358a1a0177ee935ee366df4cacaeb33bbca6e026fa8c23"} Jul 27 16:57:51.376622 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:51.376595 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d85f5b8c7-b6vcz" Jul 27 16:57:51.376754 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:51.376595 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" event={"ID":"f4626394-7b10-4288-82f6-a61e9a170203","Type":"ContainerStarted","Data":"1a1ea01c4db94a57b9ed0d118879490c6f5e387ed1b2c7240cbb2d35646d7f29"} Jul 27 16:57:51.442837 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:51.442798 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d85f5b8c7-b6vcz"] Jul 27 16:57:51.451389 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:51.451274 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d85f5b8c7-b6vcz"] Jul 27 16:57:52.064359 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:52.064322 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="160e821a-575f-4aa6-8552-0b333f613c77" path="/var/lib/kubelet/pods/160e821a-575f-4aa6-8552-0b333f613c77/volumes" Jul 27 16:57:53.388154 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:53.388113 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4scq8" event={"ID":"e328c2d0-1baf-4aba-8b4f-63c8accb0d29","Type":"ContainerStarted","Data":"f0c1b4941ff07815d5bd8bf9cfae4bf771601d3e0cab1eb0dd92676edecdae93"} Jul 27 16:57:53.679189 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:53.679081 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:53.751955 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:53.751908 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:54.066940 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:54.066864 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:54.167092 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:54.167058 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:54.232913 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:54.232846 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:54.542061 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:54.542009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:54.544864 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:54.544802 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c12e97b8-48e3-4341-a8b5-700608955f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lr9b8\" (UID: \"c12e97b8-48e3-4341-a8b5-700608955f33\") " pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:54.751617 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:54.751589 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bzm7g\"" Jul 27 16:57:54.759735 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:54.759702 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lr9b8" Jul 27 16:57:57.643763 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:57.643731 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:59.041979 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.041894 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lr9b8"] Jul 27 16:57:59.047590 ip-10-0-128-46 kubenswrapper[2557]: W0727 16:57:59.047548 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12e97b8_48e3_4341_a8b5_700608955f33.slice/crio-72e071aa76d32fd00100d65ced6e25713e3145fb73324e04ded15d30b61d2207 WatchSource:0}: Error finding container 72e071aa76d32fd00100d65ced6e25713e3145fb73324e04ded15d30b61d2207: Status 404 returned error can't find the container with id 72e071aa76d32fd00100d65ced6e25713e3145fb73324e04ded15d30b61d2207 Jul 27 16:57:59.276729 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.276701 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:59.407834 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.407803 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4scq8" event={"ID":"e328c2d0-1baf-4aba-8b4f-63c8accb0d29","Type":"ContainerStarted","Data":"2989c140ebe8751b34152ea677bc2f23173b95be659e7320521fd6d8ed89bd60"} Jul 27 16:57:59.409301 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.409270 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" event={"ID":"025239ef-c6af-4ec7-9eba-458e07a845d8","Type":"ContainerStarted","Data":"067c18db2ba98035aafe5521556f09f90eb8a064348801fdcbbdce9b4b505164"} Jul 27 16:57:59.411050 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.411023 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdkgm" event={"ID":"61ac0229-6cf5-4b4d-a406-42101ea16da7","Type":"ContainerStarted","Data":"c0e2a0ab4bc617d96b0a712494cdffd65c0e0b2d6be34a8d8ec75ef5ff8affc6"} Jul 27 16:57:59.411148 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.411057 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdkgm" event={"ID":"61ac0229-6cf5-4b4d-a406-42101ea16da7","Type":"ContainerStarted","Data":"8518c9756570b0c85c7ddf8ad62d8f17971fcf7278f596ee90c38ba6f65e67b5"} Jul 27 16:57:59.411148 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.411092 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sdkgm" Jul 27 16:57:59.413121 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.413093 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" event={"ID":"f4626394-7b10-4288-82f6-a61e9a170203","Type":"ContainerStarted","Data":"3c1ee3f5ebb7b320a74732d156f0b50947816b7ad078343865f96cdfc47c73d6"} Jul 27 16:57:59.413241 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.413127 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" event={"ID":"f4626394-7b10-4288-82f6-a61e9a170203","Type":"ContainerStarted","Data":"7cbb3afdf818837eeb62a700d7cb30d9cb0a5c01cd45575180d755211a51953b"} Jul 27 16:57:59.413241 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.413145 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" event={"ID":"f4626394-7b10-4288-82f6-a61e9a170203","Type":"ContainerStarted","Data":"77f06511f19b7ba27ae9699f500222ec404532aac93e59351fba794969914dc2"} Jul 27 16:57:59.414233 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.414204 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lr9b8" event={"ID":"c12e97b8-48e3-4341-a8b5-700608955f33","Type":"ContainerStarted","Data":"72e071aa76d32fd00100d65ced6e25713e3145fb73324e04ded15d30b61d2207"} Jul 27 16:57:59.414417 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.414396 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:59.415622 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.415595 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t54n4" event={"ID":"2402f312-cad4-44fa-a9ae-b752b73c6eb3","Type":"ContainerStarted","Data":"b00417921dd542396c655555dd02d42cbed551df703dec196caf7f6fd7137c86"} Jul 27 16:57:59.416861 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.416842 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7xqlz" event={"ID":"acf25ffa-614a-417b-a565-e97ebcc3d76c","Type":"ContainerStarted","Data":"b1a24da11aadb624f971f2ca9eb191c04dd76e71c1c49a66f3f058d7f5393cf2"} Jul 27 16:57:59.416993 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.416954 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:57:59.429006 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.428968 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4scq8" podStartSLOduration=73.079657715 podStartE2EDuration="1m15.428957065s" totalImagesPullingTime="2.34929935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:59.426763609 +0000 UTC m=+75.897089805" watchObservedRunningTime="2026-07-27 16:57:59.428957065 +0000 UTC m=+75.899283260" Jul 27 16:57:59.448193 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.448120 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t54n4" podStartSLOduration=66.696222386 podStartE2EDuration="1m15.448103513s" totalImagesPullingTime="8.751881127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:59.447663828 +0000 UTC m=+75.917990036" watchObservedRunningTime="2026-07-27 16:57:59.448103513 +0000 UTC m=+75.918429685" Jul 27 16:57:59.472771 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.472729 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7xqlz" podStartSLOduration=66.825503419 podStartE2EDuration="1m15.472714639s" totalImagesPullingTime="8.64721122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:59.471260191 +0000 UTC m=+75.941586408" watchObservedRunningTime="2026-07-27 16:57:59.472714639 +0000 UTC m=+75.943040833" Jul 27 16:57:59.498974 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.498921 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sdkgm" podStartSLOduration=33.399337687 podStartE2EDuration="41.498903045s" totalImagesPullingTime="8.099565358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:59.496889327 +0000 UTC m=+75.967215522" watchObservedRunningTime="2026-07-27 16:57:59.498903045 +0000 UTC m=+75.969229241" Jul 27 16:57:59.513108 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.513058 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f5d44f6-c7jqj" podStartSLOduration=2.013063902 podStartE2EDuration="10.513042007s" totalImagesPullingTime="8.499978105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:59.512781238 +0000 UTC m=+75.983107435" watchObservedRunningTime="2026-07-27 16:57:59.513042007 +0000 UTC m=+75.983368203" Jul 27 16:57:59.530018 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.529991 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:59.539564 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.539528 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75f4f79c8b-tl6t9" podStartSLOduration=2.041444608 podStartE2EDuration="10.539516355s" totalImagesPullingTime="8.498071747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:59.538693343 +0000 UTC m=+76.009019539" watchObservedRunningTime="2026-07-27 16:57:59.539516355 +0000 UTC m=+76.009842528" Jul 27 16:57:59.611857 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.611826 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:59.612575 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.612554 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x2tl8"] Jul 27 16:57:59.615141 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.615121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.617676 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.617657 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qbmgp\"" Jul 27 16:57:59.617770 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.617716 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 16:57:59.617826 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.617793 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 16:57:59.618248 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.618229 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 16:57:59.618381 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.618305 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 16:57:59.619287 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.619272 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 16:57:59.619439 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.619421 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 16:57:59.683849 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.683772 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:59.684810 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.684789 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-wtmp\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.684882 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.684824 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-accelerators-collector-config\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.684882 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.684851 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwprm\" (UniqueName: \"kubernetes.io/projected/315c980f-4d4d-4c47-b2b0-00a20854279e-kube-api-access-mwprm\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.684960 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.684923 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-textfile\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.684992 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.684971 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.685041 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.684999 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-root\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.685041 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.685031 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-tls\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.685108 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.685082 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-sys\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.685139 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.685109 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/315c980f-4d4d-4c47-b2b0-00a20854279e-metrics-client-ca\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.785797 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.785763 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-wtmp\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.785959 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.785800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-accelerators-collector-config\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.785959 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.785824 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwprm\" (UniqueName: \"kubernetes.io/projected/315c980f-4d4d-4c47-b2b0-00a20854279e-kube-api-access-mwprm\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.785959 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.785862 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-textfile\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.785959 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.785921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786159 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.785962 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-wtmp\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786159 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.785964 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-root\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786159 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.786009 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-root\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786159 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.786013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-tls\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786159 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.786082 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-sys\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786159 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.786108 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/315c980f-4d4d-4c47-b2b0-00a20854279e-metrics-client-ca\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786457 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.786216 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-textfile\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786457 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:59.786321 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 27 16:57:59.786457 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.786361 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/315c980f-4d4d-4c47-b2b0-00a20854279e-sys\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786457 ip-10-0-128-46 kubenswrapper[2557]: E0727 16:57:59.786380 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-tls podName:315c980f-4d4d-4c47-b2b0-00a20854279e nodeName:}" failed. No retries permitted until 2026-07-27 16:58:00.286360275 +0000 UTC m=+76.756686464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-tls") pod "node-exporter-x2tl8" (UID: "315c980f-4d4d-4c47-b2b0-00a20854279e") : secret "node-exporter-tls" not found Jul 27 16:57:59.786629 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.786579 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-accelerators-collector-config\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.786687 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.786650 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/315c980f-4d4d-4c47-b2b0-00a20854279e-metrics-client-ca\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.788372 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.788354 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.794597 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.794575 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwprm\" (UniqueName: \"kubernetes.io/projected/315c980f-4d4d-4c47-b2b0-00a20854279e-kube-api-access-mwprm\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:57:59.872107 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.872073 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:57:59.969256 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:57:59.969147 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:00.069773 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:00.069740 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:00.128488 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:00.128401 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:00.290588 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:00.290085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-tls\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:58:00.294838 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:00.294768 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/315c980f-4d4d-4c47-b2b0-00a20854279e-node-exporter-tls\") pod \"node-exporter-x2tl8\" (UID: \"315c980f-4d4d-4c47-b2b0-00a20854279e\") " pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:58:00.502932 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:00.502898 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:00.524998 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:00.524962 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x2tl8" Jul 27 16:58:01.226396 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:01.226364 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:01.289332 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:01.289302 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:01.426990 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:01.426956 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x2tl8" event={"ID":"315c980f-4d4d-4c47-b2b0-00a20854279e","Type":"ContainerStarted","Data":"6f27e48153e04e552168212ee39630a78c5857f6744cdbfc5119e418f8184559"} Jul 27 16:58:02.499118 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:02.499079 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:02.605851 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:02.605813 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:02.808267 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:02.808151 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:03.057876 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:03.057840 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:03.757263 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:03.757233 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:04.284086 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:04.284045 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:04.396000 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:04.395962 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:05.444758 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:05.444721 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lr9b8" event={"ID":"c12e97b8-48e3-4341-a8b5-700608955f33","Type":"ContainerStarted","Data":"2ec6097de9c4162d482409681d671fde46d4285bca012f038f0b123e2cc46c1c"} Jul 27 16:58:05.446401 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:05.446370 2557 generic.go:356] "Generic (PLEG): container finished" podID="315c980f-4d4d-4c47-b2b0-00a20854279e" containerID="de157695fbeacace9949eda942397642948bfb3f0ccd43899b4f94eeb8fc94f6" exitCode=0 Jul 27 16:58:05.446531 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:05.446449 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x2tl8" event={"ID":"315c980f-4d4d-4c47-b2b0-00a20854279e","Type":"ContainerDied","Data":"de157695fbeacace9949eda942397642948bfb3f0ccd43899b4f94eeb8fc94f6"} Jul 27 16:58:05.842052 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:05.841971 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:06.452652 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:06.452613 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x2tl8" event={"ID":"315c980f-4d4d-4c47-b2b0-00a20854279e","Type":"ContainerStarted","Data":"759c1009dffb9328fab0fc82288825405cc0f4815c146ef8d212d153ce429236"} Jul 27 16:58:06.453108 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:06.452673 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x2tl8" event={"ID":"315c980f-4d4d-4c47-b2b0-00a20854279e","Type":"ContainerStarted","Data":"6640bc31bf674f06f4e7768594f5aa615fa1ea386b781c3b28f243d275a87796"} Jul 27 16:58:06.473799 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:06.473745 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x2tl8" podStartSLOduration=3.153487862 podStartE2EDuration="7.473729642s" totalImagesPullingTime="4.255357961s" totalInitContainerRuntime="64.883819ms" isStatefulPod=true podCreationTimestamp="2026-07-27 16:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:06.473302583 +0000 UTC m=+82.943628779" watchObservedRunningTime="2026-07-27 16:58:06.473729642 +0000 UTC m=+82.944055836" Jul 27 16:58:09.423582 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:09.423549 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sdkgm" Jul 27 16:58:09.471029 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:09.470994 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lr9b8" event={"ID":"c12e97b8-48e3-4341-a8b5-700608955f33","Type":"ContainerStarted","Data":"1ffd4b259948317dc9af2356f94652bab1f613554d3f3e8e25621c1e906d23d7"} Jul 27 16:58:10.578238 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:10.578191 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:12.482461 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:12.482422 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lr9b8" event={"ID":"c12e97b8-48e3-4341-a8b5-700608955f33","Type":"ContainerStarted","Data":"cf321a4970f06ff5c4f7dc11a38c313eab772a4c58387a79f98d95a7c941f07b"} Jul 27 16:58:12.505846 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:12.505799 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lr9b8" podStartSLOduration=37.586779342 podStartE2EDuration="50.505780535s" totalImagesPullingTime="12.919001193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:57:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:12.505415248 +0000 UTC m=+88.975741455" watchObservedRunningTime="2026-07-27 16:58:12.505780535 +0000 UTC m=+88.976106729" Jul 27 16:58:13.793211 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:13.793159 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:14.270103 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:14.270075 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:14.333604 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:14.333576 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:14.711007 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:14.710963 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:14.812159 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:14.812134 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:15.028060 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:15.027972 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:15.099403 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:15.099378 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:15.257109 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:15.257056 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:15.489636 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:15.489604 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:16.154861 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:16.154796 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:16.210073 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:16.210042 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:16.499673 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:16.499597 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:16.705015 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:16.704981 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:16.891420 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:16.891386 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:18.597704 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:18.597673 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:18.711246 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:18.711214 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:19.287219 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:19.287163 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:19.384927 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:19.384893 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:22.727966 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:22.727935 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:22.828603 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:22.828574 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:23.048881 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:23.048792 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:23.152280 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:23.152242 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:24.172391 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:24.172357 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:24.229197 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:24.229150 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:27.587447 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:27.587415 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:28.097721 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:28.097684 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:30.364692 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:30.364656 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:30.423806 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:30.423778 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7xqlz" Jul 27 16:58:30.446443 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:30.446411 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:30.502778 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:30.502746 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:32.749882 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:32.749847 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:32.816621 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:32.816592 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:33.724942 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:33.724896 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:33.826538 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:33.826515 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:35.951336 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:35.951302 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:36.052540 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:36.052502 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:36.851080 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:36.851040 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:42.156567 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:42.156537 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2tl8_315c980f-4d4d-4c47-b2b0-00a20854279e/init-textfile/0.log" Jul 27 16:58:42.166932 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:42.166856 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:42.231711 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:42.231680 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:42.355612 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:42.355582 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2tl8_315c980f-4d4d-4c47-b2b0-00a20854279e/node-exporter/0.log" Jul 27 16:58:42.555344 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:42.555272 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2tl8_315c980f-4d4d-4c47-b2b0-00a20854279e/kube-rbac-proxy/0.log" Jul 27 16:58:43.099723 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:43.099688 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:43.284476 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:43.284439 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:43.384244 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:43.384212 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:43.485354 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:43.485326 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:50.943743 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:50.943657 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:51.004478 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:51.004440 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:59.392754 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:59.392718 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:58:59.453040 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:58:59.453010 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:18.881164 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:18.881126 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:19.231239 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:19.231145 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:27.987519 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:27.987476 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:28.086920 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:28.086877 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:28.184791 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:28.184754 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:28.736680 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:28.736647 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:28.837764 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:28.837738 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:39.272844 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:39.272759 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:39.330576 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:39.330539 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 16:59:39.452608 ip-10-0-128-46 kubenswrapper[2557]: I0727 16:59:39.452577 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:08.844605 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:08.844570 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:08.945104 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:08.945072 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:34.165428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:34.165369 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:34.226650 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:34.226623 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:45.487428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:45.487391 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:46.157913 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:46.157876 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:46.259724 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:46.259689 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:50.950560 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:50.950521 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:51.050516 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:51.050483 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:00:51.108232 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:00:51.108200 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:01:14.897781 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:01:14.897699 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:01:14.960696 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:01:14.960667 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:01:25.712071 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:01:25.712021 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:01:25.810869 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:01:25.810842 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:01:25.870288 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:01:25.870256 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:01:43.887193 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:01:43.887150 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 17:01:53.281420 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:01:53.281384 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:01:57.167445 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:01:57.167400 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:15.727139 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:15.727107 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:16.043012 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:16.042933 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:19.509801 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:19.509768 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:19.609863 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:19.609833 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:20.466330 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:20.466292 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:20.611549 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:20.611521 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:21.814139 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:21.814102 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:22.910877 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:22.910847 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:23.123229 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.123197 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:23.289632 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.289563 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t"] Jul 27 17:02:23.292409 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.292394 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" Jul 27 17:02:23.294506 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.294477 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 27 17:02:23.295009 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.294992 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bp86x\"" Jul 27 17:02:23.295009 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.295000 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 27 17:02:23.304969 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.304945 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t"] Jul 27 17:02:23.393485 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.393445 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdzdp\" (UniqueName: \"kubernetes.io/projected/4969ac33-4676-4f9f-8f60-f35a4596da78-kube-api-access-tdzdp\") pod \"cert-manager-cainjector-559cdd8c87-mvc7t\" (UID: \"4969ac33-4676-4f9f-8f60-f35a4596da78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" Jul 27 17:02:23.393485 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.393499 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4969ac33-4676-4f9f-8f60-f35a4596da78-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvc7t\" (UID: \"4969ac33-4676-4f9f-8f60-f35a4596da78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" Jul 27 17:02:23.493767 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.493734 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdzdp\" (UniqueName: \"kubernetes.io/projected/4969ac33-4676-4f9f-8f60-f35a4596da78-kube-api-access-tdzdp\") pod \"cert-manager-cainjector-559cdd8c87-mvc7t\" (UID: \"4969ac33-4676-4f9f-8f60-f35a4596da78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" Jul 27 17:02:23.493903 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.493781 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4969ac33-4676-4f9f-8f60-f35a4596da78-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvc7t\" (UID: \"4969ac33-4676-4f9f-8f60-f35a4596da78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" Jul 27 17:02:23.501286 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.501257 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4969ac33-4676-4f9f-8f60-f35a4596da78-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvc7t\" (UID: \"4969ac33-4676-4f9f-8f60-f35a4596da78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" Jul 27 17:02:23.501379 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.501348 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdzdp\" (UniqueName: \"kubernetes.io/projected/4969ac33-4676-4f9f-8f60-f35a4596da78-kube-api-access-tdzdp\") pod \"cert-manager-cainjector-559cdd8c87-mvc7t\" (UID: \"4969ac33-4676-4f9f-8f60-f35a4596da78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" Jul 27 17:02:23.600922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.600849 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" Jul 27 17:02:23.719220 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.719171 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t"] Jul 27 17:02:23.722075 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:02:23.722049 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4969ac33_4676_4f9f_8f60_f35a4596da78.slice/crio-ccf279d8f74a555d0baadf10fbe7af76af2f3c25aa61e9b3b7af11a9311e99f5 WatchSource:0}: Error finding container ccf279d8f74a555d0baadf10fbe7af76af2f3c25aa61e9b3b7af11a9311e99f5: Status 404 returned error can't find the container with id ccf279d8f74a555d0baadf10fbe7af76af2f3c25aa61e9b3b7af11a9311e99f5 Jul 27 17:02:23.723968 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:23.723951 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:02:24.268962 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:24.268928 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" event={"ID":"4969ac33-4676-4f9f-8f60-f35a4596da78","Type":"ContainerStarted","Data":"ccf279d8f74a555d0baadf10fbe7af76af2f3c25aa61e9b3b7af11a9311e99f5"} Jul 27 17:02:26.106568 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:26.106524 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:26.204465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:26.204422 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:30.290599 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:30.290563 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" event={"ID":"4969ac33-4676-4f9f-8f60-f35a4596da78","Type":"ContainerStarted","Data":"6be58d08fe6e2cff94c1a718f4728d79e28c45a65a5febba39d0449211be9f81"} Jul 27 17:02:30.308636 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:30.308586 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvc7t" podStartSLOduration=1.269662254 podStartE2EDuration="7.308569234s" totalImagesPullingTime="6.03890698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:30.307138535 +0000 UTC m=+346.777464731" watchObservedRunningTime="2026-07-27 17:02:30.308569234 +0000 UTC m=+346.778895430" Jul 27 17:02:30.426105 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:30.426077 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:30.568886 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:30.568767 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:30.659842 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:30.659793 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:31.084272 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:31.084240 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:36.520593 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:36.520492 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:36.618711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:36.618681 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:41.181077 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.181043 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4j5j9"] Jul 27 17:02:41.186704 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.186684 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4j5j9" Jul 27 17:02:41.188849 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.188827 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-96lks\"" Jul 27 17:02:41.194339 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.194313 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4j5j9"] Jul 27 17:02:41.221654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.221627 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5dm\" (UniqueName: \"kubernetes.io/projected/67868290-9f87-408d-8861-b34bcf7fc9d3-kube-api-access-zf5dm\") pod \"cert-manager-55b4d954d5-4j5j9\" (UID: \"67868290-9f87-408d-8861-b34bcf7fc9d3\") " pod="cert-manager/cert-manager-55b4d954d5-4j5j9" Jul 27 17:02:41.221764 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.221673 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67868290-9f87-408d-8861-b34bcf7fc9d3-bound-sa-token\") pod \"cert-manager-55b4d954d5-4j5j9\" (UID: \"67868290-9f87-408d-8861-b34bcf7fc9d3\") " pod="cert-manager/cert-manager-55b4d954d5-4j5j9" Jul 27 17:02:41.322078 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.322045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5dm\" (UniqueName: \"kubernetes.io/projected/67868290-9f87-408d-8861-b34bcf7fc9d3-kube-api-access-zf5dm\") pod \"cert-manager-55b4d954d5-4j5j9\" (UID: \"67868290-9f87-408d-8861-b34bcf7fc9d3\") " pod="cert-manager/cert-manager-55b4d954d5-4j5j9" Jul 27 17:02:41.322256 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.322097 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67868290-9f87-408d-8861-b34bcf7fc9d3-bound-sa-token\") pod \"cert-manager-55b4d954d5-4j5j9\" (UID: \"67868290-9f87-408d-8861-b34bcf7fc9d3\") " pod="cert-manager/cert-manager-55b4d954d5-4j5j9" Jul 27 17:02:41.330225 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.330198 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67868290-9f87-408d-8861-b34bcf7fc9d3-bound-sa-token\") pod \"cert-manager-55b4d954d5-4j5j9\" (UID: \"67868290-9f87-408d-8861-b34bcf7fc9d3\") " pod="cert-manager/cert-manager-55b4d954d5-4j5j9" Jul 27 17:02:41.330417 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.330398 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5dm\" (UniqueName: \"kubernetes.io/projected/67868290-9f87-408d-8861-b34bcf7fc9d3-kube-api-access-zf5dm\") pod \"cert-manager-55b4d954d5-4j5j9\" (UID: \"67868290-9f87-408d-8861-b34bcf7fc9d3\") " pod="cert-manager/cert-manager-55b4d954d5-4j5j9" Jul 27 17:02:41.495992 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.495908 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4j5j9" Jul 27 17:02:41.619346 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:41.619313 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4j5j9"] Jul 27 17:02:41.622075 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:02:41.622045 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67868290_9f87_408d_8861_b34bcf7fc9d3.slice/crio-19eafa65bd13b324f87735e56eb2ce5c59213ef830bb3aae04dd0b0a288834e3 WatchSource:0}: Error finding container 19eafa65bd13b324f87735e56eb2ce5c59213ef830bb3aae04dd0b0a288834e3: Status 404 returned error can't find the container with id 19eafa65bd13b324f87735e56eb2ce5c59213ef830bb3aae04dd0b0a288834e3 Jul 27 17:02:42.328584 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:42.328546 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4j5j9" event={"ID":"67868290-9f87-408d-8861-b34bcf7fc9d3","Type":"ContainerStarted","Data":"d090ac0710b7eecd9d8397d0d5b86e237f1fd4a6ba505a8ca3a59e0e7fca919d"} Jul 27 17:02:42.328584 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:42.328588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4j5j9" event={"ID":"67868290-9f87-408d-8861-b34bcf7fc9d3","Type":"ContainerStarted","Data":"19eafa65bd13b324f87735e56eb2ce5c59213ef830bb3aae04dd0b0a288834e3"} Jul 27 17:02:42.345813 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:42.345767 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4j5j9" podStartSLOduration=1.345753766 podStartE2EDuration="1.345753766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:42.344991937 +0000 UTC m=+358.815318131" watchObservedRunningTime="2026-07-27 17:02:42.345753766 +0000 UTC m=+358.816079960" Jul 27 17:02:43.657623 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:43.657589 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:43.758730 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:43.758696 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:45.338353 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:45.338315 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:57.717287 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:57.717254 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:57.821536 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:57.821508 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:58.333499 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.333456 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:58.529618 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.529565 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:58.635042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.635015 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:58.696687 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.696657 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7"] Jul 27 17:02:58.700769 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.700750 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.703273 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.703250 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 17:02:58.703388 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.703258 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 17:02:58.703680 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.703664 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h676l\"" Jul 27 17:02:58.703755 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.703664 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 17:02:58.703755 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.703664 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 17:02:58.722049 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.722024 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7"] Jul 27 17:02:58.827004 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.826974 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:58.847733 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.847707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-tmp\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.847818 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.847759 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62w6\" (UniqueName: \"kubernetes.io/projected/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-kube-api-access-q62w6\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.847818 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.847792 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.847818 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.847814 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.847981 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.847830 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-manifests\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.948978 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.948909 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-tmp\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.948978 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.948952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62w6\" (UniqueName: \"kubernetes.io/projected/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-kube-api-access-q62w6\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.949130 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.948980 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.949130 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.949013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.949130 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.949039 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-manifests\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.949426 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.949400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-manifests\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.951158 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.951129 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-tmp\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.951374 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.951354 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.951506 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.951487 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.959901 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.959878 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62w6\" (UniqueName: \"kubernetes.io/projected/969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1-kube-api-access-q62w6\") pod \"opendatahub-operator-controller-manager-565c484996-lshg7\" (UID: \"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:58.990480 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:58.990450 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:59.011129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:59.011098 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:02:59.096843 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:59.096810 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:59.181137 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:59.181002 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7"] Jul 27 17:02:59.183865 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:02:59.183842 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969a1945_f7c9_4ffa_bfa0_2c508d6bfaf1.slice/crio-0ccfb1dbf849bea8677156a6fe5e1b1e7a25f9347ddb58ca4eec2b5f0fab5dcb WatchSource:0}: Error finding container 0ccfb1dbf849bea8677156a6fe5e1b1e7a25f9347ddb58ca4eec2b5f0fab5dcb: Status 404 returned error can't find the container with id 0ccfb1dbf849bea8677156a6fe5e1b1e7a25f9347ddb58ca4eec2b5f0fab5dcb Jul 27 17:02:59.192222 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:59.192197 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:59.262495 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:59.262385 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:59.359255 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:59.359210 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:02:59.384245 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:59.384216 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" event={"ID":"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1","Type":"ContainerStarted","Data":"0ccfb1dbf849bea8677156a6fe5e1b1e7a25f9347ddb58ca4eec2b5f0fab5dcb"} Jul 27 17:02:59.815010 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:02:59.814978 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:03.398785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:03.398745 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" event={"ID":"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1","Type":"ContainerStarted","Data":"5096f328333bf3edb5910300f66425c2e54cda82810c341a64d506775e629696"} Jul 27 17:03:04.403256 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:04.403216 2557 generic.go:356] "Generic (PLEG): container finished" podID="969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1" containerID="5096f328333bf3edb5910300f66425c2e54cda82810c341a64d506775e629696" exitCode=0 Jul 27 17:03:04.403647 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:04.403278 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" event={"ID":"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1","Type":"ContainerDied","Data":"5096f328333bf3edb5910300f66425c2e54cda82810c341a64d506775e629696"} Jul 27 17:03:05.408279 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:05.408242 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" event={"ID":"969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1","Type":"ContainerStarted","Data":"45c4131979f086ab02b249236475cb1f7ea92ef779d34c52b3dea482b70b4887"} Jul 27 17:03:05.408675 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:05.408361 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:03:05.432500 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:05.432448 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" podStartSLOduration=2.762262643 podStartE2EDuration="7.432432336s" totalImagesPullingTime="3.578210302s" totalInitContainerRuntime="1.091959391s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:05.430617685 +0000 UTC m=+381.900943881" watchObservedRunningTime="2026-07-27 17:03:05.432432336 +0000 UTC m=+381.902758598" Jul 27 17:03:14.274198 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:14.274108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:14.414816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:14.414779 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:16.413998 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:16.413962 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-lshg7" Jul 27 17:03:19.884584 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:19.884542 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:19.984577 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:19.984543 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:20.611020 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.610984 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pj8fh"] Jul 27 17:03:20.626142 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.626107 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pj8fh"] Jul 27 17:03:20.626357 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.626248 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:20.628576 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.628554 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hkhx8\"" Jul 27 17:03:20.628686 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.628611 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 27 17:03:20.652941 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.652910 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:20.718219 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.718161 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwjr\" (UniqueName: \"kubernetes.io/projected/2109efaa-eafa-4c8b-a66c-98c97c873333-kube-api-access-tnwjr\") pod \"odh-model-controller-7998c67dc4-pj8fh\" (UID: \"2109efaa-eafa-4c8b-a66c-98c97c873333\") " pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:20.718382 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.718257 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2109efaa-eafa-4c8b-a66c-98c97c873333-cert\") pod \"odh-model-controller-7998c67dc4-pj8fh\" (UID: \"2109efaa-eafa-4c8b-a66c-98c97c873333\") " pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:20.753747 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.753715 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:20.819471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.819437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwjr\" (UniqueName: \"kubernetes.io/projected/2109efaa-eafa-4c8b-a66c-98c97c873333-kube-api-access-tnwjr\") pod \"odh-model-controller-7998c67dc4-pj8fh\" (UID: \"2109efaa-eafa-4c8b-a66c-98c97c873333\") " pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:20.819471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.819480 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2109efaa-eafa-4c8b-a66c-98c97c873333-cert\") pod \"odh-model-controller-7998c67dc4-pj8fh\" (UID: \"2109efaa-eafa-4c8b-a66c-98c97c873333\") " pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:20.821956 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.821932 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2109efaa-eafa-4c8b-a66c-98c97c873333-cert\") pod \"odh-model-controller-7998c67dc4-pj8fh\" (UID: \"2109efaa-eafa-4c8b-a66c-98c97c873333\") " pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:20.828031 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.828003 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwjr\" (UniqueName: \"kubernetes.io/projected/2109efaa-eafa-4c8b-a66c-98c97c873333-kube-api-access-tnwjr\") pod \"odh-model-controller-7998c67dc4-pj8fh\" (UID: \"2109efaa-eafa-4c8b-a66c-98c97c873333\") " pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:20.853027 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.852998 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:20.939227 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:20.939191 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:21.068623 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.068592 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pj8fh"] Jul 27 17:03:21.072196 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:03:21.072145 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2109efaa_eafa_4c8b_a66c_98c97c873333.slice/crio-88f70194b39b5fbf59c891d4ee360630f6dd847194771b69add8f8945ba260ae WatchSource:0}: Error finding container 88f70194b39b5fbf59c891d4ee360630f6dd847194771b69add8f8945ba260ae: Status 404 returned error can't find the container with id 88f70194b39b5fbf59c891d4ee360630f6dd847194771b69add8f8945ba260ae Jul 27 17:03:21.157089 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.157050 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z8zwd"] Jul 27 17:03:21.160896 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.160868 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:21.164857 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.164838 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 27 17:03:21.169723 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.169702 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:21.193059 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.192980 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z8zwd"] Jul 27 17:03:21.322281 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.322247 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzcn\" (UniqueName: \"kubernetes.io/projected/879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5-kube-api-access-2kzcn\") pod \"model-serving-api-54d89bd5fc-z8zwd\" (UID: \"879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:21.322443 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.322291 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5-tls-certs\") pod \"model-serving-api-54d89bd5fc-z8zwd\" (UID: \"879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:21.423674 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.423641 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzcn\" (UniqueName: \"kubernetes.io/projected/879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5-kube-api-access-2kzcn\") pod \"model-serving-api-54d89bd5fc-z8zwd\" (UID: \"879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:21.423674 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.423678 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5-tls-certs\") pod \"model-serving-api-54d89bd5fc-z8zwd\" (UID: \"879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:21.426006 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.425983 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5-tls-certs\") pod \"model-serving-api-54d89bd5fc-z8zwd\" (UID: \"879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:21.431161 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.431139 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:21.432055 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.432030 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzcn\" (UniqueName: \"kubernetes.io/projected/879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5-kube-api-access-2kzcn\") pod \"model-serving-api-54d89bd5fc-z8zwd\" (UID: \"879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:21.460775 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.460709 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" event={"ID":"2109efaa-eafa-4c8b-a66c-98c97c873333","Type":"ContainerStarted","Data":"88f70194b39b5fbf59c891d4ee360630f6dd847194771b69add8f8945ba260ae"} Jul 27 17:03:21.470963 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.470941 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:21.599717 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:21.599690 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z8zwd"] Jul 27 17:03:21.601907 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:03:21.601877 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879bf6a2_16bf_40e9_8cc2_87e4eef8b7e5.slice/crio-0a462e39a99f4a5fd7f2ee12b38398ea11a78909149cb2b738e212c8618cb5ac WatchSource:0}: Error finding container 0a462e39a99f4a5fd7f2ee12b38398ea11a78909149cb2b738e212c8618cb5ac: Status 404 returned error can't find the container with id 0a462e39a99f4a5fd7f2ee12b38398ea11a78909149cb2b738e212c8618cb5ac Jul 27 17:03:22.341263 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:22.341221 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:22.465981 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:22.465943 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" event={"ID":"879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5","Type":"ContainerStarted","Data":"0a462e39a99f4a5fd7f2ee12b38398ea11a78909149cb2b738e212c8618cb5ac"} Jul 27 17:03:24.479527 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:24.479494 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:24.688340 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:24.688303 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:24.788517 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:24.788434 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:25.480721 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:25.480606 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" event={"ID":"879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5","Type":"ContainerStarted","Data":"13c0bdd2c83b2f5cfd0c60509f803d642e9c76608d8c657b1f47e4528f7cded2"} Jul 27 17:03:25.481205 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:25.480767 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:25.482321 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:25.482288 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" event={"ID":"2109efaa-eafa-4c8b-a66c-98c97c873333","Type":"ContainerStarted","Data":"3a304eaf26ca6a44a87cf00b3705a87859cd1b9c7f4d8656414be81fc74f1e13"} Jul 27 17:03:25.482449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:25.482436 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:25.497700 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:25.497660 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" podStartSLOduration=0.893826213 podStartE2EDuration="4.497647958s" totalImagesPullingTime="3.603821745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:25.496368802 +0000 UTC m=+401.966694996" watchObservedRunningTime="2026-07-27 17:03:25.497647958 +0000 UTC m=+401.967974146" Jul 27 17:03:25.513947 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:25.513895 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" podStartSLOduration=1.377266363 podStartE2EDuration="5.51387618s" totalImagesPullingTime="4.136609817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:25.513118992 +0000 UTC m=+401.983445188" watchObservedRunningTime="2026-07-27 17:03:25.51387618 +0000 UTC m=+401.984202376" Jul 27 17:03:30.917569 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.917536 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9"] Jul 27 17:03:30.920364 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.920347 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:30.924637 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.924588 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-49kzh\"" Jul 27 17:03:30.924777 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.924625 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 17:03:30.943576 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.943487 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z"] Jul 27 17:03:30.946800 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.946778 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9"] Jul 27 17:03:30.946920 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.946884 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:30.952340 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.952318 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4g4qb\"" Jul 27 17:03:30.956055 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.956038 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 17:03:30.976155 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:30.976134 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z"] Jul 27 17:03:31.000217 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.000171 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da8ded80-b230-44e4-97c6-59f49d5e4c56-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s5z5z\" (UID: \"da8ded80-b230-44e4-97c6-59f49d5e4c56\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:31.000364 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.000243 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gml8f\" (UniqueName: \"kubernetes.io/projected/da8ded80-b230-44e4-97c6-59f49d5e4c56-kube-api-access-gml8f\") pod \"llmisvc-controller-manager-777c6bd7d5-s5z5z\" (UID: \"da8ded80-b230-44e4-97c6-59f49d5e4c56\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:31.093210 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.092396 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:31.101577 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.101544 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da8ded80-b230-44e4-97c6-59f49d5e4c56-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s5z5z\" (UID: \"da8ded80-b230-44e4-97c6-59f49d5e4c56\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:31.101743 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.101608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gml8f\" (UniqueName: \"kubernetes.io/projected/da8ded80-b230-44e4-97c6-59f49d5e4c56-kube-api-access-gml8f\") pod \"llmisvc-controller-manager-777c6bd7d5-s5z5z\" (UID: \"da8ded80-b230-44e4-97c6-59f49d5e4c56\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:31.101743 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.101637 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4nmp\" (UniqueName: \"kubernetes.io/projected/39fd0a82-20a5-4271-90f7-a3fcbccc0d29-kube-api-access-p4nmp\") pod \"kserve-controller-manager-d6f4b65b8-d2wq9\" (UID: \"39fd0a82-20a5-4271-90f7-a3fcbccc0d29\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:31.101743 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.101723 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39fd0a82-20a5-4271-90f7-a3fcbccc0d29-cert\") pod \"kserve-controller-manager-d6f4b65b8-d2wq9\" (UID: \"39fd0a82-20a5-4271-90f7-a3fcbccc0d29\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:31.102018 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:03:31.101995 2557 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 27 17:03:31.102087 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:03:31.102068 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8ded80-b230-44e4-97c6-59f49d5e4c56-cert podName:da8ded80-b230-44e4-97c6-59f49d5e4c56 nodeName:}" failed. No retries permitted until 2026-07-27 17:03:31.602047798 +0000 UTC m=+408.072373987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da8ded80-b230-44e4-97c6-59f49d5e4c56-cert") pod "llmisvc-controller-manager-777c6bd7d5-s5z5z" (UID: "da8ded80-b230-44e4-97c6-59f49d5e4c56") : secret "llmisvc-webhook-server-cert" not found Jul 27 17:03:31.124402 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.124367 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh"] Jul 27 17:03:31.125483 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.125462 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gml8f\" (UniqueName: \"kubernetes.io/projected/da8ded80-b230-44e4-97c6-59f49d5e4c56-kube-api-access-gml8f\") pod \"llmisvc-controller-manager-777c6bd7d5-s5z5z\" (UID: \"da8ded80-b230-44e4-97c6-59f49d5e4c56\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:31.127980 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.127958 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.138838 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.138814 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 17:03:31.138937 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.138851 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 17:03:31.138937 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.138869 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mdvcg\"" Jul 27 17:03:31.143212 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.143155 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 17:03:31.143420 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.143407 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 17:03:31.177074 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.177006 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh"] Jul 27 17:03:31.201996 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.201968 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7dn9\" (UniqueName: \"kubernetes.io/projected/bef354c2-610b-453c-a659-5075fb9afeb7-kube-api-access-f7dn9\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.202123 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.202000 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4nmp\" (UniqueName: \"kubernetes.io/projected/39fd0a82-20a5-4271-90f7-a3fcbccc0d29-kube-api-access-p4nmp\") pod \"kserve-controller-manager-d6f4b65b8-d2wq9\" (UID: \"39fd0a82-20a5-4271-90f7-a3fcbccc0d29\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:31.202123 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.202027 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef354c2-610b-453c-a659-5075fb9afeb7-tls-certs\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.202123 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.202043 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bef354c2-610b-453c-a659-5075fb9afeb7-tmp\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.202302 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.202201 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39fd0a82-20a5-4271-90f7-a3fcbccc0d29-cert\") pod \"kserve-controller-manager-d6f4b65b8-d2wq9\" (UID: \"39fd0a82-20a5-4271-90f7-a3fcbccc0d29\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:31.204711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.204688 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39fd0a82-20a5-4271-90f7-a3fcbccc0d29-cert\") pod \"kserve-controller-manager-d6f4b65b8-d2wq9\" (UID: \"39fd0a82-20a5-4271-90f7-a3fcbccc0d29\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:31.234378 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.234350 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4nmp\" (UniqueName: \"kubernetes.io/projected/39fd0a82-20a5-4271-90f7-a3fcbccc0d29-kube-api-access-p4nmp\") pod \"kserve-controller-manager-d6f4b65b8-d2wq9\" (UID: \"39fd0a82-20a5-4271-90f7-a3fcbccc0d29\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:31.235406 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.235386 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:31.302656 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.302628 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef354c2-610b-453c-a659-5075fb9afeb7-tls-certs\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.302768 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.302658 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bef354c2-610b-453c-a659-5075fb9afeb7-tmp\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.302768 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.302723 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7dn9\" (UniqueName: \"kubernetes.io/projected/bef354c2-610b-453c-a659-5075fb9afeb7-kube-api-access-f7dn9\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.304806 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.304778 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bef354c2-610b-453c-a659-5075fb9afeb7-tmp\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.305047 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.305027 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef354c2-610b-453c-a659-5075fb9afeb7-tls-certs\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.307570 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.307549 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:31.314866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.314846 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7dn9\" (UniqueName: \"kubernetes.io/projected/bef354c2-610b-453c-a659-5075fb9afeb7-kube-api-access-f7dn9\") pod \"kube-auth-proxy-5cc796b767-hvjkh\" (UID: \"bef354c2-610b-453c-a659-5075fb9afeb7\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.441708 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.441639 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" Jul 27 17:03:31.531481 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.531445 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:31.567691 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.567617 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh"] Jul 27 17:03:31.570960 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:03:31.570932 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef354c2_610b_453c_a659_5075fb9afeb7.slice/crio-010482f8eb1d2d36657bdcd6b95fd3ec332d5b4844985d6d97ac986532527410 WatchSource:0}: Error finding container 010482f8eb1d2d36657bdcd6b95fd3ec332d5b4844985d6d97ac986532527410: Status 404 returned error can't find the container with id 010482f8eb1d2d36657bdcd6b95fd3ec332d5b4844985d6d97ac986532527410 Jul 27 17:03:31.604586 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.604558 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da8ded80-b230-44e4-97c6-59f49d5e4c56-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s5z5z\" (UID: \"da8ded80-b230-44e4-97c6-59f49d5e4c56\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:31.606770 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.606749 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da8ded80-b230-44e4-97c6-59f49d5e4c56-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s5z5z\" (UID: \"da8ded80-b230-44e4-97c6-59f49d5e4c56\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:31.658373 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.658343 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9"] Jul 27 17:03:31.660540 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:03:31.660516 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fd0a82_20a5_4271_90f7_a3fcbccc0d29.slice/crio-f146ff7776060923ea3ce12c632ddaf5f28600745d78b7d50e8c03512d785084 WatchSource:0}: Error finding container f146ff7776060923ea3ce12c632ddaf5f28600745d78b7d50e8c03512d785084: Status 404 returned error can't find the container with id f146ff7776060923ea3ce12c632ddaf5f28600745d78b7d50e8c03512d785084 Jul 27 17:03:31.856905 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:31.856833 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:32.011492 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.011462 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z"] Jul 27 17:03:32.014034 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:03:32.014007 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda8ded80_b230_44e4_97c6_59f49d5e4c56.slice/crio-c26236e18432450d379504e8dc66e21508555659b6e26c963facdf6d4a50ee5d WatchSource:0}: Error finding container c26236e18432450d379504e8dc66e21508555659b6e26c963facdf6d4a50ee5d: Status 404 returned error can't find the container with id c26236e18432450d379504e8dc66e21508555659b6e26c963facdf6d4a50ee5d Jul 27 17:03:32.183155 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.183121 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:32.282168 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.282137 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:32.388732 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.388697 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:32.513809 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.513724 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" event={"ID":"bef354c2-610b-453c-a659-5075fb9afeb7","Type":"ContainerStarted","Data":"010482f8eb1d2d36657bdcd6b95fd3ec332d5b4844985d6d97ac986532527410"} Jul 27 17:03:32.515251 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.515218 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" event={"ID":"da8ded80-b230-44e4-97c6-59f49d5e4c56","Type":"ContainerStarted","Data":"c26236e18432450d379504e8dc66e21508555659b6e26c963facdf6d4a50ee5d"} Jul 27 17:03:32.516440 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.516416 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" event={"ID":"39fd0a82-20a5-4271-90f7-a3fcbccc0d29","Type":"ContainerStarted","Data":"f146ff7776060923ea3ce12c632ddaf5f28600745d78b7d50e8c03512d785084"} Jul 27 17:03:32.858827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.858750 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz"] Jul 27 17:03:32.860143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.860097 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:32.862510 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.862486 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:32.866060 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.866039 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 27 17:03:32.866349 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.866329 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 27 17:03:32.866457 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.866389 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 27 17:03:32.866457 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.866441 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4rt4m\"" Jul 27 17:03:32.866703 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.866688 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 17:03:32.882445 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.882420 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz"] Jul 27 17:03:32.921221 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.921168 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:32.921373 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.921230 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ec4f2f3-2370-4089-9299-993253494855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:32.921373 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.921258 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec4f2f3-2370-4089-9299-993253494855-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:32.921373 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.921297 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:32.921503 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.921376 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2x6b\" (UniqueName: \"kubernetes.io/projected/4ec4f2f3-2370-4089-9299-993253494855-kube-api-access-d2x6b\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:32.921503 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.921426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:32.921503 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.921452 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec4f2f3-2370-4089-9299-993253494855-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:32.949084 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:32.949053 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:33.021868 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.021827 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2x6b\" (UniqueName: \"kubernetes.io/projected/4ec4f2f3-2370-4089-9299-993253494855-kube-api-access-d2x6b\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.023262 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.021887 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.023262 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.021932 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec4f2f3-2370-4089-9299-993253494855-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.023262 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.022003 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.023262 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.022032 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ec4f2f3-2370-4089-9299-993253494855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.023262 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.022067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec4f2f3-2370-4089-9299-993253494855-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.023262 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.022103 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.023727 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.023394 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ec4f2f3-2370-4089-9299-993253494855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.025053 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.025026 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.025204 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.025166 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.025627 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.025604 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ec4f2f3-2370-4089-9299-993253494855-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.026982 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.026957 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec4f2f3-2370-4089-9299-993253494855-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.048419 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.048386 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec4f2f3-2370-4089-9299-993253494855-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.049737 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.049681 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2x6b\" (UniqueName: \"kubernetes.io/projected/4ec4f2f3-2370-4089-9299-993253494855-kube-api-access-d2x6b\") pod \"istiod-openshift-gateway-5677ccb49b-wwxbz\" (UID: \"4ec4f2f3-2370-4089-9299-993253494855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.182271 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.181973 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:33.425605 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:33.425569 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz"] Jul 27 17:03:33.537525 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:03:33.537459 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec4f2f3_2370_4089_9299_993253494855.slice/crio-5342b6e13e8d73fc7b92c46b4ff005f1d3e12028d881b5eb89cacff7b83706ec WatchSource:0}: Error finding container 5342b6e13e8d73fc7b92c46b4ff005f1d3e12028d881b5eb89cacff7b83706ec: Status 404 returned error can't find the container with id 5342b6e13e8d73fc7b92c46b4ff005f1d3e12028d881b5eb89cacff7b83706ec Jul 27 17:03:34.279625 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:34.279588 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:34.530640 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:34.530562 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" event={"ID":"4ec4f2f3-2370-4089-9299-993253494855","Type":"ContainerStarted","Data":"5342b6e13e8d73fc7b92c46b4ff005f1d3e12028d881b5eb89cacff7b83706ec"} Jul 27 17:03:35.538574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:35.538433 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" event={"ID":"39fd0a82-20a5-4271-90f7-a3fcbccc0d29","Type":"ContainerStarted","Data":"fa01058d4dc8d2422077ee838d7198e1162c00c125e13b276ae7e47af287b027"} Jul 27 17:03:35.538574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:35.538539 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:03:35.541494 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:35.541263 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" event={"ID":"bef354c2-610b-453c-a659-5075fb9afeb7","Type":"ContainerStarted","Data":"acf672bd9e022a8c666b4185b60b1ead6bbe54961ad613a00bc8b89fbc7beb72"} Jul 27 17:03:35.557410 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:35.557365 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" podStartSLOduration=2.084073004 podStartE2EDuration="5.557345558s" totalImagesPullingTime="3.473272554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:35.55613813 +0000 UTC m=+412.026464327" watchObservedRunningTime="2026-07-27 17:03:35.557345558 +0000 UTC m=+412.027671753" Jul 27 17:03:35.577434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:35.577373 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cc796b767-hvjkh" podStartSLOduration=1.016506595 podStartE2EDuration="4.577359374s" totalImagesPullingTime="3.560852779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:35.576116061 +0000 UTC m=+412.046442251" watchObservedRunningTime="2026-07-27 17:03:35.577359374 +0000 UTC m=+412.047685571" Jul 27 17:03:36.489216 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:36.489168 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pj8fh" Jul 27 17:03:36.491816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:36.491794 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-z8zwd" Jul 27 17:03:37.508406 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:37.508377 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:03:37.508652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:37.508433 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:03:38.473060 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.473011 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:38.556809 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.556771 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" event={"ID":"da8ded80-b230-44e4-97c6-59f49d5e4c56","Type":"ContainerStarted","Data":"62567d034c33d61ce0e18715df2ec1059f41df72dabf5ba05c40632952176fbb"} Jul 27 17:03:38.557317 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.557232 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:03:38.558968 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.558936 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" event={"ID":"4ec4f2f3-2370-4089-9299-993253494855","Type":"ContainerStarted","Data":"24bb1f740f61c1ee6269b66456464404d26ad223f92bea33735f63f98a07f8b8"} Jul 27 17:03:38.559223 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.559201 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:38.561103 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.561077 2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-wwxbz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 17:03:38.561229 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.561127 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" podUID="4ec4f2f3-2370-4089-9299-993253494855" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:03:38.572156 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.572134 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:38.576619 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.576573 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" podStartSLOduration=3.086589019 podStartE2EDuration="8.576556853s" totalImagesPullingTime="5.489967834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:38.575122731 +0000 UTC m=+415.045448941" watchObservedRunningTime="2026-07-27 17:03:38.576556853 +0000 UTC m=+415.046883049" Jul 27 17:03:38.599537 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:38.599471 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" podStartSLOduration=2.630800592 podStartE2EDuration="6.599444216s" totalImagesPullingTime="3.968643624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:38.5982931 +0000 UTC m=+415.068619298" watchObservedRunningTime="2026-07-27 17:03:38.599444216 +0000 UTC m=+415.069770411" Jul 27 17:03:39.445606 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:39.445570 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:39.564722 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:39.564693 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wwxbz" Jul 27 17:03:39.671566 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:39.671532 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:03:39.907440 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:03:39.907405 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:06.551821 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:06.551738 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d2wq9" Jul 27 17:04:09.566237 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:09.566206 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s5z5z" Jul 27 17:04:21.586713 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:21.586673 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:21.692029 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:21.691994 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:21.780761 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:21.780728 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:44.903037 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:44.903003 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:44.958818 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:44.958769 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:53.250400 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:53.250364 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:55.829858 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:55.829825 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:56.033903 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:56.033870 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:56.134809 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:56.134778 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:56.686448 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:56.686413 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:04:57.547316 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:04:57.547282 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:01.055082 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:01.055037 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:01.521352 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:01.521321 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:01.614863 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:01.614830 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:03.320863 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:03.320834 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:04.735744 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:04.735708 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:05.711826 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:05.711795 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:06.330684 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:06.330641 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:09.344046 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:09.344012 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:09.446952 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:09.446920 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:14.665257 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:14.665222 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:14.763670 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:14.763635 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:15.063592 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:15.063508 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:15.168734 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:15.168704 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:17.932239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:17.932170 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:18.537279 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:18.537249 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:18.864767 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:18.864685 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:18.964884 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:18.964851 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:21.229252 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:21.229216 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:21.325491 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:21.325450 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:27.892569 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:27.892535 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld"] Jul 27 17:05:27.894730 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:27.894714 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:27.898720 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:27.898693 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rs4tp\"" Jul 27 17:05:27.898839 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:27.898700 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 17:05:27.899205 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:27.899170 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 17:05:27.911021 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:27.911001 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld"] Jul 27 17:05:28.009598 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.009555 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3fd533-332e-4fa2-b919-fc42639c9c2e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" (UID: \"da3fd533-332e-4fa2-b919-fc42639c9c2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:28.009598 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.009594 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznhx\" (UniqueName: \"kubernetes.io/projected/da3fd533-332e-4fa2-b919-fc42639c9c2e-kube-api-access-hznhx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" (UID: \"da3fd533-332e-4fa2-b919-fc42639c9c2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:28.110279 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.110239 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3fd533-332e-4fa2-b919-fc42639c9c2e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" (UID: \"da3fd533-332e-4fa2-b919-fc42639c9c2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:28.110279 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.110280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznhx\" (UniqueName: \"kubernetes.io/projected/da3fd533-332e-4fa2-b919-fc42639c9c2e-kube-api-access-hznhx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" (UID: \"da3fd533-332e-4fa2-b919-fc42639c9c2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:28.110697 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.110672 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3fd533-332e-4fa2-b919-fc42639c9c2e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" (UID: \"da3fd533-332e-4fa2-b919-fc42639c9c2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:28.119432 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.119407 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznhx\" (UniqueName: \"kubernetes.io/projected/da3fd533-332e-4fa2-b919-fc42639c9c2e-kube-api-access-hznhx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" (UID: \"da3fd533-332e-4fa2-b919-fc42639c9c2e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:28.183523 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.183443 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:28.204348 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.204324 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:28.286007 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.285963 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:28.338761 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.338728 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld"] Jul 27 17:05:28.342418 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:05:28.342393 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3fd533_332e_4fa2_b919_fc42639c9c2e.slice/crio-a67ef4576fbf433ded9c7671dc28731cebebb5fc682cd3fd8aab9e1ee693d9ab WatchSource:0}: Error finding container a67ef4576fbf433ded9c7671dc28731cebebb5fc682cd3fd8aab9e1ee693d9ab: Status 404 returned error can't find the container with id a67ef4576fbf433ded9c7671dc28731cebebb5fc682cd3fd8aab9e1ee693d9ab Jul 27 17:05:28.503681 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.503598 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld"] Jul 27 17:05:28.516229 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.516204 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld"] Jul 27 17:05:28.541498 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.541467 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk"] Jul 27 17:05:28.544818 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.544802 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:28.574030 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.574000 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk"] Jul 27 17:05:28.615592 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.615552 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pqcsk\" (UID: \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:28.615750 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.615673 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxzvg\" (UniqueName: \"kubernetes.io/projected/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-kube-api-access-qxzvg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pqcsk\" (UID: \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:28.716904 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.716869 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxzvg\" (UniqueName: \"kubernetes.io/projected/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-kube-api-access-qxzvg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pqcsk\" (UID: \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:28.717091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.716917 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pqcsk\" (UID: \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:28.717334 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.717310 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pqcsk\" (UID: \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:28.727803 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.727780 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxzvg\" (UniqueName: \"kubernetes.io/projected/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-kube-api-access-qxzvg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pqcsk\" (UID: \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:28.854396 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:28.854294 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:29.005866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:29.005837 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk"] Jul 27 17:05:29.009981 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:05:29.009950 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1dc03cb_6d8b_45af_a6ae_119bf1403ebe.slice/crio-67ebc29a4023901ee210621a19e51445a8bfa25a7b2b3d8b1cfc04affd5b8898 WatchSource:0}: Error finding container 67ebc29a4023901ee210621a19e51445a8bfa25a7b2b3d8b1cfc04affd5b8898: Status 404 returned error can't find the container with id 67ebc29a4023901ee210621a19e51445a8bfa25a7b2b3d8b1cfc04affd5b8898 Jul 27 17:05:29.993367 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:29.993323 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" event={"ID":"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe","Type":"ContainerStarted","Data":"67ebc29a4023901ee210621a19e51445a8bfa25a7b2b3d8b1cfc04affd5b8898"} Jul 27 17:05:33.262380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:33.262340 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:33.361764 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:33.361728 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:33.859401 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:33.859360 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:34.207042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:34.207008 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:35.016319 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.016259 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" containerName="manager" containerID="cri-o://8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae" gracePeriod=2 Jul 27 17:05:35.017902 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.017877 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" event={"ID":"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe","Type":"ContainerStarted","Data":"54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e"} Jul 27 17:05:35.018034 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.018015 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:35.018152 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:05:35.018134 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" is forbidden: User \"system:node:ip-10-0-128-46.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-46.ec2.internal' and this object" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:35.040509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.040460 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" podStartSLOduration=2.074115889 podStartE2EDuration="7.040445311s" totalImagesPullingTime="4.966329422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:05:35.039078087 +0000 UTC m=+531.509404282" watchObservedRunningTime="2026-07-27 17:05:35.040445311 +0000 UTC m=+531.510771577" Jul 27 17:05:35.041640 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:05:35.041612 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" is forbidden: User \"system:node:ip-10-0-128-46.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-46.ec2.internal' and this object" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:35.257857 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.257833 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:35.259747 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:05:35.259721 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" is forbidden: User \"system:node:ip-10-0-128-46.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-46.ec2.internal' and this object" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:35.373338 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.373230 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hznhx\" (UniqueName: \"kubernetes.io/projected/da3fd533-332e-4fa2-b919-fc42639c9c2e-kube-api-access-hznhx\") pod \"da3fd533-332e-4fa2-b919-fc42639c9c2e\" (UID: \"da3fd533-332e-4fa2-b919-fc42639c9c2e\") " Jul 27 17:05:35.373338 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.373334 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3fd533-332e-4fa2-b919-fc42639c9c2e-extensions-socket-volume\") pod \"da3fd533-332e-4fa2-b919-fc42639c9c2e\" (UID: \"da3fd533-332e-4fa2-b919-fc42639c9c2e\") " Jul 27 17:05:35.373614 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.373588 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3fd533-332e-4fa2-b919-fc42639c9c2e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "da3fd533-332e-4fa2-b919-fc42639c9c2e" (UID: "da3fd533-332e-4fa2-b919-fc42639c9c2e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:05:35.375504 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.375482 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3fd533-332e-4fa2-b919-fc42639c9c2e-kube-api-access-hznhx" (OuterVolumeSpecName: "kube-api-access-hznhx") pod "da3fd533-332e-4fa2-b919-fc42639c9c2e" (UID: "da3fd533-332e-4fa2-b919-fc42639c9c2e"). InnerVolumeSpecName "kube-api-access-hznhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:05:35.474807 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.474731 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3fd533-332e-4fa2-b919-fc42639c9c2e-extensions-socket-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:05:35.474807 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:35.474760 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznhx\" (UniqueName: \"kubernetes.io/projected/da3fd533-332e-4fa2-b919-fc42639c9c2e-kube-api-access-hznhx\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:05:36.022737 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:36.022702 2557 generic.go:356] "Generic (PLEG): container finished" podID="da3fd533-332e-4fa2-b919-fc42639c9c2e" containerID="8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae" exitCode=2 Jul 27 17:05:36.023215 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:36.022745 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:36.023215 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:36.022806 2557 scope.go:117] "RemoveContainer" containerID="8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae" Jul 27 17:05:36.025164 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:05:36.025140 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" is forbidden: User \"system:node:ip-10-0-128-46.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-46.ec2.internal' and this object" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:36.032588 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:36.032564 2557 scope.go:117] "RemoveContainer" containerID="8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae" Jul 27 17:05:36.032848 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:05:36.032827 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae\": container with ID starting with 8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae not found: ID does not exist" containerID="8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae" Jul 27 17:05:36.032900 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:36.032857 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae"} err="failed to get container status \"8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae\": rpc error: code = NotFound desc = could not find container \"8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae\": container with ID starting with 8c76a7c7503887dca2403b4b3d92d2e4596a4168d858931ce0244c79377f5bae not found: ID does not exist" Jul 27 17:05:36.033863 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:05:36.033841 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-f9xld\" is forbidden: User \"system:node:ip-10-0-128-46.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-46.ec2.internal' and this object" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f9xld" Jul 27 17:05:36.064666 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:36.064630 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" path="/var/lib/kubelet/pods/da3fd533-332e-4fa2-b919-fc42639c9c2e/volumes" Jul 27 17:05:37.261664 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:37.261627 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:37.677774 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:37.677726 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:41.077612 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:41.077580 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:41.151668 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:41.151638 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:05:46.025833 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:46.025802 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:59.015606 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.015567 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk"] Jul 27 17:05:59.016145 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.015857 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" podUID="b1dc03cb-6d8b-45af-a6ae-119bf1403ebe" containerName="manager" containerID="cri-o://54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e" gracePeriod=10 Jul 27 17:05:59.268741 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.268684 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:05:59.352435 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.352392 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxzvg\" (UniqueName: \"kubernetes.io/projected/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-kube-api-access-qxzvg\") pod \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\" (UID: \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\") " Jul 27 17:05:59.352605 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.352465 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-extensions-socket-volume\") pod \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\" (UID: \"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe\") " Jul 27 17:05:59.352834 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.352810 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b1dc03cb-6d8b-45af-a6ae-119bf1403ebe" (UID: "b1dc03cb-6d8b-45af-a6ae-119bf1403ebe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:05:59.354571 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.354546 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-kube-api-access-qxzvg" (OuterVolumeSpecName: "kube-api-access-qxzvg") pod "b1dc03cb-6d8b-45af-a6ae-119bf1403ebe" (UID: "b1dc03cb-6d8b-45af-a6ae-119bf1403ebe"). InnerVolumeSpecName "kube-api-access-qxzvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:05:59.453126 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.453089 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxzvg\" (UniqueName: \"kubernetes.io/projected/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-kube-api-access-qxzvg\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:05:59.453126 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.453118 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe-extensions-socket-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:05:59.795888 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:05:59.795854 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:00.119661 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.119626 2557 generic.go:356] "Generic (PLEG): container finished" podID="b1dc03cb-6d8b-45af-a6ae-119bf1403ebe" containerID="54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e" exitCode=0 Jul 27 17:06:00.120041 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.119676 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" event={"ID":"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe","Type":"ContainerDied","Data":"54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e"} Jul 27 17:06:00.120041 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.119721 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" event={"ID":"b1dc03cb-6d8b-45af-a6ae-119bf1403ebe","Type":"ContainerDied","Data":"67ebc29a4023901ee210621a19e51445a8bfa25a7b2b3d8b1cfc04affd5b8898"} Jul 27 17:06:00.120041 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.119744 2557 scope.go:117] "RemoveContainer" containerID="54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e" Jul 27 17:06:00.120041 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.119678 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk" Jul 27 17:06:00.128342 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.128322 2557 scope.go:117] "RemoveContainer" containerID="54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e" Jul 27 17:06:00.128700 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:06:00.128676 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e\": container with ID starting with 54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e not found: ID does not exist" containerID="54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e" Jul 27 17:06:00.128757 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.128711 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e"} err="failed to get container status \"54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e\": rpc error: code = NotFound desc = could not find container \"54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e\": container with ID starting with 54b89803118642fa55861065ce49d60225a9ef178a879793d76c0b1b816a986e not found: ID does not exist" Jul 27 17:06:00.187847 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.187814 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk"] Jul 27 17:06:00.193093 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:00.193064 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pqcsk"] Jul 27 17:06:02.065990 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.065952 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1dc03cb-6d8b-45af-a6ae-119bf1403ebe" path="/var/lib/kubelet/pods/b1dc03cb-6d8b-45af-a6ae-119bf1403ebe/volumes" Jul 27 17:06:02.574615 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.574580 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z"] Jul 27 17:06:02.574881 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.574869 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" containerName="manager" Jul 27 17:06:02.574936 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.574883 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" containerName="manager" Jul 27 17:06:02.574936 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.574898 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1dc03cb-6d8b-45af-a6ae-119bf1403ebe" containerName="manager" Jul 27 17:06:02.574936 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.574903 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1dc03cb-6d8b-45af-a6ae-119bf1403ebe" containerName="manager" Jul 27 17:06:02.575041 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.574950 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1dc03cb-6d8b-45af-a6ae-119bf1403ebe" containerName="manager" Jul 27 17:06:02.575041 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.574957 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3fd533-332e-4fa2-b919-fc42639c9c2e" containerName="manager" Jul 27 17:06:02.577850 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.577830 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.589013 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.588985 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vq8h8\"" Jul 27 17:06:02.601403 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.601377 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z"] Jul 27 17:06:02.627869 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.627837 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:02.678380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678337 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.678380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.678711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678428 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.678711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678475 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.678711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678519 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.678711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678550 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.678711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678574 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.678711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678604 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slh6c\" (UniqueName: \"kubernetes.io/projected/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-kube-api-access-slh6c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.678711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.678621 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779427 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779388 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779427 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779427 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779660 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779450 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779660 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779476 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slh6c\" (UniqueName: \"kubernetes.io/projected/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-kube-api-access-slh6c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779660 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779660 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779541 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779660 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779660 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779609 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779951 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779795 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.779951 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779892 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.780042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.779988 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.780089 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.780032 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.780089 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.780064 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.780513 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.780490 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.781809 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.781785 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.782015 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.781998 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.794861 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.794829 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.796087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.796062 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slh6c\" (UniqueName: \"kubernetes.io/projected/8264e4fa-d20a-4d04-9fd2-384a1cc596b6-kube-api-access-slh6c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d7q5z\" (UID: \"8264e4fa-d20a-4d04-9fd2-384a1cc596b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:02.856445 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.856357 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:02.888665 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:02.888621 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:03.035234 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:03.035199 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z"] Jul 27 17:06:03.038514 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:06:03.038488 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8264e4fa_d20a_4d04_9fd2_384a1cc596b6.slice/crio-ab8315c871f2ad8fda66f45bfe24da42ba07b54de9e0d1ab434e2262b5c3fb2d WatchSource:0}: Error finding container ab8315c871f2ad8fda66f45bfe24da42ba07b54de9e0d1ab434e2262b5c3fb2d: Status 404 returned error can't find the container with id ab8315c871f2ad8fda66f45bfe24da42ba07b54de9e0d1ab434e2262b5c3fb2d Jul 27 17:06:03.141479 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:03.141438 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" event={"ID":"8264e4fa-d20a-4d04-9fd2-384a1cc596b6","Type":"ContainerStarted","Data":"ab8315c871f2ad8fda66f45bfe24da42ba07b54de9e0d1ab434e2262b5c3fb2d"} Jul 27 17:06:06.216443 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:06.216402 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:06:06.216804 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:06.216489 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:06:06.216804 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:06.216543 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:06:07.160121 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:07.160084 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" event={"ID":"8264e4fa-d20a-4d04-9fd2-384a1cc596b6","Type":"ContainerStarted","Data":"a97769fd41eaa0434709ef9e78e47d4f5680206811fd2820b96414b631b086fd"} Jul 27 17:06:07.192156 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:07.192103 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" podStartSLOduration=2.01648727 podStartE2EDuration="5.192085967s" totalImagesPullingTime="3.175598697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:07.190982701 +0000 UTC m=+563.661308897" watchObservedRunningTime="2026-07-27 17:06:07.192085967 +0000 UTC m=+563.662412161" Jul 27 17:06:07.888982 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:07.888941 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:07.893559 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:07.893532 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:08.170058 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.169966 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:08.171013 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.170987 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:06:08.237840 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.237800 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:08.238028 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.237977 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gsx5q"] Jul 27 17:06:08.241075 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.241054 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:08.243997 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.243974 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 17:06:08.244339 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.244324 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qmxv5\"" Jul 27 17:06:08.244418 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.244335 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 17:06:08.245801 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.245784 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 17:06:08.253243 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.253219 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gsx5q"] Jul 27 17:06:08.319936 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.319900 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gsx5q"] Jul 27 17:06:08.326368 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.326338 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mg6k\" (UniqueName: \"kubernetes.io/projected/7177be43-e995-4d56-99bc-7c490f0d99e7-kube-api-access-2mg6k\") pod \"limitador-limitador-64c94fd64f-gsx5q\" (UID: \"7177be43-e995-4d56-99bc-7c490f0d99e7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:08.326518 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.326433 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7177be43-e995-4d56-99bc-7c490f0d99e7-config-file\") pod \"limitador-limitador-64c94fd64f-gsx5q\" (UID: \"7177be43-e995-4d56-99bc-7c490f0d99e7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:08.326642 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.326621 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:08.391091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.391058 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:08.427056 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.426962 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mg6k\" (UniqueName: \"kubernetes.io/projected/7177be43-e995-4d56-99bc-7c490f0d99e7-kube-api-access-2mg6k\") pod \"limitador-limitador-64c94fd64f-gsx5q\" (UID: \"7177be43-e995-4d56-99bc-7c490f0d99e7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:08.427255 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.427061 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7177be43-e995-4d56-99bc-7c490f0d99e7-config-file\") pod \"limitador-limitador-64c94fd64f-gsx5q\" (UID: \"7177be43-e995-4d56-99bc-7c490f0d99e7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:08.427751 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.427728 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7177be43-e995-4d56-99bc-7c490f0d99e7-config-file\") pod \"limitador-limitador-64c94fd64f-gsx5q\" (UID: \"7177be43-e995-4d56-99bc-7c490f0d99e7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:08.444236 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.444211 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mg6k\" (UniqueName: \"kubernetes.io/projected/7177be43-e995-4d56-99bc-7c490f0d99e7-kube-api-access-2mg6k\") pod \"limitador-limitador-64c94fd64f-gsx5q\" (UID: \"7177be43-e995-4d56-99bc-7c490f0d99e7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:08.552118 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.552079 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:08.691313 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:08.691109 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gsx5q"] Jul 27 17:06:08.693793 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:06:08.693763 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7177be43_e995_4d56_99bc_7c490f0d99e7.slice/crio-debe6f0a839c5c903245385906694b55eac0ac2de1b47864bfbe541c994a473c WatchSource:0}: Error finding container debe6f0a839c5c903245385906694b55eac0ac2de1b47864bfbe541c994a473c: Status 404 returned error can't find the container with id debe6f0a839c5c903245385906694b55eac0ac2de1b47864bfbe541c994a473c Jul 27 17:06:09.134618 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:09.134583 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:09.175059 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:09.175022 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" event={"ID":"7177be43-e995-4d56-99bc-7c490f0d99e7","Type":"ContainerStarted","Data":"debe6f0a839c5c903245385906694b55eac0ac2de1b47864bfbe541c994a473c"} Jul 27 17:06:09.214099 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:09.214063 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:12.190169 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:12.190134 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" event={"ID":"7177be43-e995-4d56-99bc-7c490f0d99e7","Type":"ContainerStarted","Data":"b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e"} Jul 27 17:06:12.190572 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:12.190253 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:12.221162 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:12.221118 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" podStartSLOduration=1.574646319 podStartE2EDuration="4.221102617s" totalImagesPullingTime="2.646456298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:12.220921202 +0000 UTC m=+568.691247400" watchObservedRunningTime="2026-07-27 17:06:12.221102617 +0000 UTC m=+568.691428812" Jul 27 17:06:14.632465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:14.632422 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:14.737756 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:14.737719 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:23.195550 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:23.195517 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:24.101205 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.101157 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gsx5q"] Jul 27 17:06:24.101383 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.101348 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" podUID="7177be43-e995-4d56-99bc-7c490f0d99e7" containerName="limitador" containerID="cri-o://b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e" gracePeriod=30 Jul 27 17:06:24.646870 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.646848 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:24.768564 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.768485 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mg6k\" (UniqueName: \"kubernetes.io/projected/7177be43-e995-4d56-99bc-7c490f0d99e7-kube-api-access-2mg6k\") pod \"7177be43-e995-4d56-99bc-7c490f0d99e7\" (UID: \"7177be43-e995-4d56-99bc-7c490f0d99e7\") " Jul 27 17:06:24.768564 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.768529 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7177be43-e995-4d56-99bc-7c490f0d99e7-config-file\") pod \"7177be43-e995-4d56-99bc-7c490f0d99e7\" (UID: \"7177be43-e995-4d56-99bc-7c490f0d99e7\") " Jul 27 17:06:24.768922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.768901 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7177be43-e995-4d56-99bc-7c490f0d99e7-config-file" (OuterVolumeSpecName: "config-file") pod "7177be43-e995-4d56-99bc-7c490f0d99e7" (UID: "7177be43-e995-4d56-99bc-7c490f0d99e7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:06:24.770531 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.770495 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7177be43-e995-4d56-99bc-7c490f0d99e7-kube-api-access-2mg6k" (OuterVolumeSpecName: "kube-api-access-2mg6k") pod "7177be43-e995-4d56-99bc-7c490f0d99e7" (UID: "7177be43-e995-4d56-99bc-7c490f0d99e7"). InnerVolumeSpecName "kube-api-access-2mg6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:06:24.869609 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.869573 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mg6k\" (UniqueName: \"kubernetes.io/projected/7177be43-e995-4d56-99bc-7c490f0d99e7-kube-api-access-2mg6k\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:06:24.869609 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:24.869601 2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7177be43-e995-4d56-99bc-7c490f0d99e7-config-file\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:06:25.244473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.244436 2557 generic.go:356] "Generic (PLEG): container finished" podID="7177be43-e995-4d56-99bc-7c490f0d99e7" containerID="b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e" exitCode=0 Jul 27 17:06:25.244642 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.244477 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" Jul 27 17:06:25.244642 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.244523 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" event={"ID":"7177be43-e995-4d56-99bc-7c490f0d99e7","Type":"ContainerDied","Data":"b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e"} Jul 27 17:06:25.244642 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.244579 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gsx5q" event={"ID":"7177be43-e995-4d56-99bc-7c490f0d99e7","Type":"ContainerDied","Data":"debe6f0a839c5c903245385906694b55eac0ac2de1b47864bfbe541c994a473c"} Jul 27 17:06:25.244642 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.244610 2557 scope.go:117] "RemoveContainer" containerID="b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e" Jul 27 17:06:25.253546 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.253529 2557 scope.go:117] "RemoveContainer" containerID="b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e" Jul 27 17:06:25.253809 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:06:25.253791 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e\": container with ID starting with b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e not found: ID does not exist" containerID="b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e" Jul 27 17:06:25.253860 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.253818 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e"} err="failed to get container status \"b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e\": rpc error: code = NotFound desc = could not find container \"b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e\": container with ID starting with b04dfe6d0be0e824954de26908e6fddeb073a84d4cbeb864655294609c31b85e not found: ID does not exist" Jul 27 17:06:25.267842 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.267819 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gsx5q"] Jul 27 17:06:25.277544 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:25.277522 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gsx5q"] Jul 27 17:06:26.063502 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:26.063468 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7177be43-e995-4d56-99bc-7c490f0d99e7" path="/var/lib/kubelet/pods/7177be43-e995-4d56-99bc-7c490f0d99e7/volumes" Jul 27 17:06:29.506652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.506615 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:29.630009 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.629958 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:29.935312 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.935275 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk"] Jul 27 17:06:29.935598 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.935586 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7177be43-e995-4d56-99bc-7c490f0d99e7" containerName="limitador" Jul 27 17:06:29.935640 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.935600 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7177be43-e995-4d56-99bc-7c490f0d99e7" containerName="limitador" Jul 27 17:06:29.935672 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.935653 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7177be43-e995-4d56-99bc-7c490f0d99e7" containerName="limitador" Jul 27 17:06:29.939807 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.939785 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:29.951952 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.951929 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 27 17:06:29.959595 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.959568 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dmp8m\"" Jul 27 17:06:29.960480 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.960455 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk"] Jul 27 17:06:29.960628 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.960615 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 27 17:06:29.969767 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:29.969747 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 17:06:30.012690 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.012657 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8qqr\" (UniqueName: \"kubernetes.io/projected/5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05-kube-api-access-j8qqr\") pod \"postgres-69fb4594cf-pcpqk\" (UID: \"5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:30.012846 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.012709 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05-data\") pod \"postgres-69fb4594cf-pcpqk\" (UID: \"5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:30.020316 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.020268 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:30.113467 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.113433 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8qqr\" (UniqueName: \"kubernetes.io/projected/5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05-kube-api-access-j8qqr\") pod \"postgres-69fb4594cf-pcpqk\" (UID: \"5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:30.113662 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.113490 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05-data\") pod \"postgres-69fb4594cf-pcpqk\" (UID: \"5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:30.113902 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.113883 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05-data\") pod \"postgres-69fb4594cf-pcpqk\" (UID: \"5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:30.125906 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.125877 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8qqr\" (UniqueName: \"kubernetes.io/projected/5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05-kube-api-access-j8qqr\") pod \"postgres-69fb4594cf-pcpqk\" (UID: \"5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:30.249488 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.249399 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:30.385385 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:30.385363 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk"] Jul 27 17:06:30.387642 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:06:30.387618 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1f7668_1bcb_4a9f_ad3a_b87c0d8bcc05.slice/crio-c3189e6873b0209be68bca278bb5427801a74b1f21e93a44e8d04ea226f71b87 WatchSource:0}: Error finding container c3189e6873b0209be68bca278bb5427801a74b1f21e93a44e8d04ea226f71b87: Status 404 returned error can't find the container with id c3189e6873b0209be68bca278bb5427801a74b1f21e93a44e8d04ea226f71b87 Jul 27 17:06:31.270603 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:31.270563 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" event={"ID":"5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05","Type":"ContainerStarted","Data":"c3189e6873b0209be68bca278bb5427801a74b1f21e93a44e8d04ea226f71b87"} Jul 27 17:06:31.606234 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:31.606125 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:32.059628 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:32.059596 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:39.312147 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:39.312106 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" event={"ID":"5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05","Type":"ContainerStarted","Data":"87b450958f85afc39215ebee6fc52c7b815f01b959b4ab2ae8a3eaa84664dddc"} Jul 27 17:06:39.312574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:39.312238 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:39.332972 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:39.332923 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" podStartSLOduration=1.5096932060000001 podStartE2EDuration="10.332907552s" totalImagesPullingTime="8.823214346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:39.331978974 +0000 UTC m=+595.802305171" watchObservedRunningTime="2026-07-27 17:06:39.332907552 +0000 UTC m=+595.803233747" Jul 27 17:06:45.344809 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:45.344772 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pcpqk" Jul 27 17:06:45.759643 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:45.759608 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:46.044689 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:46.044612 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:51.408954 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:51.408922 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:55.718672 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.718634 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:55.791211 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.791166 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86b57d8d46-wmbq6"] Jul 27 17:06:55.794538 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.794521 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:55.798266 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.798240 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 17:06:55.799346 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.799313 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h2wn4\"" Jul 27 17:06:55.799459 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.799419 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 17:06:55.821525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.821494 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert\") pod \"maas-controller-86b57d8d46-wmbq6\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:55.821679 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.821533 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8st\" (UniqueName: \"kubernetes.io/projected/81733de7-af5f-4b73-85f2-7655631aefea-kube-api-access-mg8st\") pod \"maas-controller-86b57d8d46-wmbq6\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:55.824069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.824042 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86b57d8d46-wmbq6"] Jul 27 17:06:55.922242 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.922203 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert\") pod \"maas-controller-86b57d8d46-wmbq6\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:55.922406 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.922254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8st\" (UniqueName: \"kubernetes.io/projected/81733de7-af5f-4b73-85f2-7655631aefea-kube-api-access-mg8st\") pod \"maas-controller-86b57d8d46-wmbq6\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:55.922406 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:06:55.922375 2557 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 27 17:06:55.922483 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:06:55.922463 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert podName:81733de7-af5f-4b73-85f2-7655631aefea nodeName:}" failed. No retries permitted until 2026-07-27 17:06:56.422440855 +0000 UTC m=+612.892767041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert") pod "maas-controller-86b57d8d46-wmbq6" (UID: "81733de7-af5f-4b73-85f2-7655631aefea") : secret "maas-controller-webhook-cert" not found Jul 27 17:06:55.942364 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.942335 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8st\" (UniqueName: \"kubernetes.io/projected/81733de7-af5f-4b73-85f2-7655631aefea-kube-api-access-mg8st\") pod \"maas-controller-86b57d8d46-wmbq6\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:55.956824 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:55.956787 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:56.067586 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.067506 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-s76xr"] Jul 27 17:06:56.070855 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.070836 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:56.087445 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.087420 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-s76xr"] Jul 27 17:06:56.100432 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.100399 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:56.123983 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.123955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert\") pod \"maas-controller-74cbc56ff7-s76xr\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:56.124144 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.124045 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6t5\" (UniqueName: \"kubernetes.io/projected/5a758cfc-217f-4cf2-b494-19a57a5a81c5-kube-api-access-xv6t5\") pod \"maas-controller-74cbc56ff7-s76xr\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:56.224681 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.224644 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert\") pod \"maas-controller-74cbc56ff7-s76xr\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:56.224876 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.224735 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6t5\" (UniqueName: \"kubernetes.io/projected/5a758cfc-217f-4cf2-b494-19a57a5a81c5-kube-api-access-xv6t5\") pod \"maas-controller-74cbc56ff7-s76xr\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:56.224876 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:06:56.224805 2557 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 27 17:06:56.224976 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:06:56.224883 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert podName:5a758cfc-217f-4cf2-b494-19a57a5a81c5 nodeName:}" failed. No retries permitted until 2026-07-27 17:06:56.724864731 +0000 UTC m=+613.195190904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert") pod "maas-controller-74cbc56ff7-s76xr" (UID: "5a758cfc-217f-4cf2-b494-19a57a5a81c5") : secret "maas-controller-webhook-cert" not found Jul 27 17:06:56.241915 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.241886 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6t5\" (UniqueName: \"kubernetes.io/projected/5a758cfc-217f-4cf2-b494-19a57a5a81c5-kube-api-access-xv6t5\") pod \"maas-controller-74cbc56ff7-s76xr\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:56.373752 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.373659 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:06:56.426985 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.426953 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert\") pod \"maas-controller-86b57d8d46-wmbq6\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:56.429279 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.429259 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert\") pod \"maas-controller-86b57d8d46-wmbq6\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:56.706288 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.706249 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:06:56.729639 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.729610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert\") pod \"maas-controller-74cbc56ff7-s76xr\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:56.731870 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.731851 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert\") pod \"maas-controller-74cbc56ff7-s76xr\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:56.849029 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.848921 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86b57d8d46-wmbq6"] Jul 27 17:06:56.851750 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:06:56.851719 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81733de7_af5f_4b73_85f2_7655631aefea.slice/crio-a8d9878bf6e943151d641e9798a3c436a4c4af2718502eedfa68324cfb1e9875 WatchSource:0}: Error finding container a8d9878bf6e943151d641e9798a3c436a4c4af2718502eedfa68324cfb1e9875: Status 404 returned error can't find the container with id a8d9878bf6e943151d641e9798a3c436a4c4af2718502eedfa68324cfb1e9875 Jul 27 17:06:56.981589 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:56.981503 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:06:57.121782 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:57.121751 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-s76xr"] Jul 27 17:06:57.127376 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:06:57.127352 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a758cfc_217f_4cf2_b494_19a57a5a81c5.slice/crio-9e83188ac3553f67787e05c3dcd9e100efbfbb6d850a623ae7ff6861fd04ca7d WatchSource:0}: Error finding container 9e83188ac3553f67787e05c3dcd9e100efbfbb6d850a623ae7ff6861fd04ca7d: Status 404 returned error can't find the container with id 9e83188ac3553f67787e05c3dcd9e100efbfbb6d850a623ae7ff6861fd04ca7d Jul 27 17:06:57.386758 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:57.386723 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" event={"ID":"5a758cfc-217f-4cf2-b494-19a57a5a81c5","Type":"ContainerStarted","Data":"9e83188ac3553f67787e05c3dcd9e100efbfbb6d850a623ae7ff6861fd04ca7d"} Jul 27 17:06:57.387803 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:57.387779 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" event={"ID":"81733de7-af5f-4b73-85f2-7655631aefea","Type":"ContainerStarted","Data":"a8d9878bf6e943151d641e9798a3c436a4c4af2718502eedfa68324cfb1e9875"} Jul 27 17:06:59.854700 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:06:59.854664 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:00.404381 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:00.404345 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" event={"ID":"5a758cfc-217f-4cf2-b494-19a57a5a81c5","Type":"ContainerStarted","Data":"c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6"} Jul 27 17:07:00.404571 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:00.404469 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:07:00.405791 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:00.405770 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" event={"ID":"81733de7-af5f-4b73-85f2-7655631aefea","Type":"ContainerStarted","Data":"3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c"} Jul 27 17:07:00.405906 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:00.405855 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:07:00.496340 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:00.496276 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" podStartSLOduration=1.975931055 podStartE2EDuration="4.496257086s" totalImagesPullingTime="2.520326031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:07:00.496203893 +0000 UTC m=+616.966530087" watchObservedRunningTime="2026-07-27 17:07:00.496257086 +0000 UTC m=+616.966583282" Jul 27 17:07:00.685243 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:00.685084 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" podStartSLOduration=2.895764465 podStartE2EDuration="5.68506262s" totalImagesPullingTime="2.789298155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:07:00.682684202 +0000 UTC m=+617.153010399" watchObservedRunningTime="2026-07-27 17:07:00.68506262 +0000 UTC m=+617.155388815" Jul 27 17:07:01.831315 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:01.831283 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:11.412031 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.411958 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:07:11.412031 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.412029 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:07:11.519714 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.519682 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86b57d8d46-wmbq6"] Jul 27 17:07:11.519913 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.519891 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" podUID="81733de7-af5f-4b73-85f2-7655631aefea" containerName="manager" containerID="cri-o://3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c" gracePeriod=30 Jul 27 17:07:11.762239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.762213 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:07:11.857859 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.857828 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert\") pod \"81733de7-af5f-4b73-85f2-7655631aefea\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " Jul 27 17:07:11.858052 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.857904 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg8st\" (UniqueName: \"kubernetes.io/projected/81733de7-af5f-4b73-85f2-7655631aefea-kube-api-access-mg8st\") pod \"81733de7-af5f-4b73-85f2-7655631aefea\" (UID: \"81733de7-af5f-4b73-85f2-7655631aefea\") " Jul 27 17:07:11.859992 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.859965 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81733de7-af5f-4b73-85f2-7655631aefea-kube-api-access-mg8st" (OuterVolumeSpecName: "kube-api-access-mg8st") pod "81733de7-af5f-4b73-85f2-7655631aefea" (UID: "81733de7-af5f-4b73-85f2-7655631aefea"). InnerVolumeSpecName "kube-api-access-mg8st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:07:11.860115 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.860095 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "81733de7-af5f-4b73-85f2-7655631aefea" (UID: "81733de7-af5f-4b73-85f2-7655631aefea"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:07:11.958840 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.958761 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg8st\" (UniqueName: \"kubernetes.io/projected/81733de7-af5f-4b73-85f2-7655631aefea-kube-api-access-mg8st\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:07:11.958840 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:11.958792 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81733de7-af5f-4b73-85f2-7655631aefea-webhook-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:07:12.455540 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.455506 2557 generic.go:356] "Generic (PLEG): container finished" podID="81733de7-af5f-4b73-85f2-7655631aefea" containerID="3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c" exitCode=0 Jul 27 17:07:12.455954 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.455553 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" Jul 27 17:07:12.455954 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.455571 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" event={"ID":"81733de7-af5f-4b73-85f2-7655631aefea","Type":"ContainerDied","Data":"3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c"} Jul 27 17:07:12.455954 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.455612 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86b57d8d46-wmbq6" event={"ID":"81733de7-af5f-4b73-85f2-7655631aefea","Type":"ContainerDied","Data":"a8d9878bf6e943151d641e9798a3c436a4c4af2718502eedfa68324cfb1e9875"} Jul 27 17:07:12.455954 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.455633 2557 scope.go:117] "RemoveContainer" containerID="3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c" Jul 27 17:07:12.464226 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.464199 2557 scope.go:117] "RemoveContainer" containerID="3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c" Jul 27 17:07:12.464461 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:07:12.464443 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c\": container with ID starting with 3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c not found: ID does not exist" containerID="3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c" Jul 27 17:07:12.464532 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.464469 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c"} err="failed to get container status \"3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c\": rpc error: code = NotFound desc = could not find container \"3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c\": container with ID starting with 3bb662e82afcc094ab72dfc717082bd2cf628a0b5b81d4b80ae8ffa97d3bc90c not found: ID does not exist" Jul 27 17:07:12.473781 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.473760 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86b57d8d46-wmbq6"] Jul 27 17:07:12.483072 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:12.483046 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86b57d8d46-wmbq6"] Jul 27 17:07:14.064097 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:14.064068 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81733de7-af5f-4b73-85f2-7655631aefea" path="/var/lib/kubelet/pods/81733de7-af5f-4b73-85f2-7655631aefea/volumes" Jul 27 17:07:29.797495 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:29.797451 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:31.765334 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:31.765293 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:31.848669 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:31.848625 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:32.269509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:32.269468 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:32.572962 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:32.572892 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:33.008441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.008405 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:33.119314 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.119280 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:33.213106 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.213072 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc"] Jul 27 17:07:33.213453 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.213439 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81733de7-af5f-4b73-85f2-7655631aefea" containerName="manager" Jul 27 17:07:33.213453 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.213454 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="81733de7-af5f-4b73-85f2-7655631aefea" containerName="manager" Jul 27 17:07:33.213541 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.213498 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="81733de7-af5f-4b73-85f2-7655631aefea" containerName="manager" Jul 27 17:07:33.218341 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.218322 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:33.223081 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.223050 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 27 17:07:33.223081 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.223073 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-929gr\"" Jul 27 17:07:33.235249 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.235226 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc"] Jul 27 17:07:33.306926 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.306850 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vcgrx"] Jul 27 17:07:33.309406 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.309390 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:33.330006 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.329982 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vcgrx"] Jul 27 17:07:33.337523 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.337500 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chkfq\" (UniqueName: \"kubernetes.io/projected/7105d860-e49f-470e-a9ee-a0773eee1584-kube-api-access-chkfq\") pod \"payload-pre-processing-687dbb6cf6-bjtcc\" (UID: \"7105d860-e49f-470e-a9ee-a0773eee1584\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:33.337650 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.337532 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7105d860-e49f-470e-a9ee-a0773eee1584-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-bjtcc\" (UID: \"7105d860-e49f-470e-a9ee-a0773eee1584\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:33.438360 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.438320 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2l9\" (UniqueName: \"kubernetes.io/projected/69dfa449-4401-4cf9-8374-0f08e94eeeae-kube-api-access-xt2l9\") pod \"payload-processing-5cc6cd444b-vcgrx\" (UID: \"69dfa449-4401-4cf9-8374-0f08e94eeeae\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:33.438525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.438371 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69dfa449-4401-4cf9-8374-0f08e94eeeae-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vcgrx\" (UID: \"69dfa449-4401-4cf9-8374-0f08e94eeeae\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:33.438525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.438438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chkfq\" (UniqueName: \"kubernetes.io/projected/7105d860-e49f-470e-a9ee-a0773eee1584-kube-api-access-chkfq\") pod \"payload-pre-processing-687dbb6cf6-bjtcc\" (UID: \"7105d860-e49f-470e-a9ee-a0773eee1584\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:33.438525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.438469 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7105d860-e49f-470e-a9ee-a0773eee1584-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-bjtcc\" (UID: \"7105d860-e49f-470e-a9ee-a0773eee1584\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:33.439275 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.439254 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7105d860-e49f-470e-a9ee-a0773eee1584-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-bjtcc\" (UID: \"7105d860-e49f-470e-a9ee-a0773eee1584\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:33.448667 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.448642 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chkfq\" (UniqueName: \"kubernetes.io/projected/7105d860-e49f-470e-a9ee-a0773eee1584-kube-api-access-chkfq\") pod \"payload-pre-processing-687dbb6cf6-bjtcc\" (UID: \"7105d860-e49f-470e-a9ee-a0773eee1584\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:33.527994 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.527959 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:33.539000 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.538969 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2l9\" (UniqueName: \"kubernetes.io/projected/69dfa449-4401-4cf9-8374-0f08e94eeeae-kube-api-access-xt2l9\") pod \"payload-processing-5cc6cd444b-vcgrx\" (UID: \"69dfa449-4401-4cf9-8374-0f08e94eeeae\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:33.539113 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.539015 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69dfa449-4401-4cf9-8374-0f08e94eeeae-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vcgrx\" (UID: \"69dfa449-4401-4cf9-8374-0f08e94eeeae\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:33.539840 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.539818 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69dfa449-4401-4cf9-8374-0f08e94eeeae-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vcgrx\" (UID: \"69dfa449-4401-4cf9-8374-0f08e94eeeae\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:33.553321 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.553295 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2l9\" (UniqueName: \"kubernetes.io/projected/69dfa449-4401-4cf9-8374-0f08e94eeeae-kube-api-access-xt2l9\") pod \"payload-processing-5cc6cd444b-vcgrx\" (UID: \"69dfa449-4401-4cf9-8374-0f08e94eeeae\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:33.619589 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.619550 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:33.674887 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.674807 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc"] Jul 27 17:07:33.682452 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.682426 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:07:33.983398 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:33.983373 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vcgrx"] Jul 27 17:07:33.985197 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:07:33.985156 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dfa449_4401_4cf9_8374_0f08e94eeeae.slice/crio-3f11ee599003c70f5d65c05c78589149987d1c3499af9458f31aa00d010d4e8f WatchSource:0}: Error finding container 3f11ee599003c70f5d65c05c78589149987d1c3499af9458f31aa00d010d4e8f: Status 404 returned error can't find the container with id 3f11ee599003c70f5d65c05c78589149987d1c3499af9458f31aa00d010d4e8f Jul 27 17:07:34.576004 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:34.575962 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" event={"ID":"69dfa449-4401-4cf9-8374-0f08e94eeeae","Type":"ContainerStarted","Data":"3f11ee599003c70f5d65c05c78589149987d1c3499af9458f31aa00d010d4e8f"} Jul 27 17:07:34.577602 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:34.577572 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" event={"ID":"7105d860-e49f-470e-a9ee-a0773eee1584","Type":"ContainerStarted","Data":"ec3ffba0b98ea4d341b2f5b3dedb21f87ff43bc12c3de37c8f645caab39df19e"} Jul 27 17:07:36.589776 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:36.589734 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" event={"ID":"7105d860-e49f-470e-a9ee-a0773eee1584","Type":"ContainerStarted","Data":"fba5dc6bcac23c8959dd4800c4e37ee43c36d8863f752399c45367548c5f7b50"} Jul 27 17:07:36.590264 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:36.589812 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:07:36.591365 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:36.591336 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" event={"ID":"69dfa449-4401-4cf9-8374-0f08e94eeeae","Type":"ContainerStarted","Data":"84106cdae14d1d6104e203bb435fac842cc799cbb4ec435952c3b1f02564f79e"} Jul 27 17:07:36.591473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:36.591393 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:36.618545 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:36.618501 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" podStartSLOduration=1.337341624 podStartE2EDuration="3.618486906s" totalImagesPullingTime="2.281145282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:07:36.617502095 +0000 UTC m=+653.087828291" watchObservedRunningTime="2026-07-27 17:07:36.618486906 +0000 UTC m=+653.088813100" Jul 27 17:07:36.646378 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:36.646325 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" podStartSLOduration=1.653491016 podStartE2EDuration="3.646307514s" totalImagesPullingTime="1.992816498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:07:36.643605483 +0000 UTC m=+653.113931677" watchObservedRunningTime="2026-07-27 17:07:36.646307514 +0000 UTC m=+653.116633710" Jul 27 17:07:39.211387 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:39.211347 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d7q5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 17:07:39.211799 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:39.211406 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" podUID="8264e4fa-d20a-4d04-9fd2-384a1cc596b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 17:07:41.616925 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:41.616890 2557 generic.go:356] "Generic (PLEG): container finished" podID="8264e4fa-d20a-4d04-9fd2-384a1cc596b6" containerID="a97769fd41eaa0434709ef9e78e47d4f5680206811fd2820b96414b631b086fd" exitCode=0 Jul 27 17:07:41.617367 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:41.616965 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" event={"ID":"8264e4fa-d20a-4d04-9fd2-384a1cc596b6","Type":"ContainerDied","Data":"a97769fd41eaa0434709ef9e78e47d4f5680206811fd2820b96414b631b086fd"} Jul 27 17:07:41.617435 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:41.617417 2557 scope.go:117] "RemoveContainer" containerID="a97769fd41eaa0434709ef9e78e47d4f5680206811fd2820b96414b631b086fd" Jul 27 17:07:41.618407 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:41.618376 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:07:41.618509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:41.618443 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:07:41.618509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:41.618477 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:07:42.623647 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:42.623602 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" event={"ID":"8264e4fa-d20a-4d04-9fd2-384a1cc596b6","Type":"ContainerStarted","Data":"90ba3305cadb63c80b2c4f78734b7b2cb37d6b4be87ae662e028f59668e2d633"} Jul 27 17:07:42.889767 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:42.889736 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:07:42.890005 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:42.889987 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d7q5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 27 17:07:42.890063 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:42.890030 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" podUID="8264e4fa-d20a-4d04-9fd2-384a1cc596b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 27 17:07:43.654987 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:43.654932 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:43.744405 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:43.744372 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:07:43.890035 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:43.890002 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d7q5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 27 17:07:43.890212 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:43.890080 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" podUID="8264e4fa-d20a-4d04-9fd2-384a1cc596b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 27 17:07:44.889559 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:44.889517 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d7q5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 27 17:07:44.889964 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:44.889577 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" podUID="8264e4fa-d20a-4d04-9fd2-384a1cc596b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 27 17:07:45.889456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:45.889413 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d7q5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 27 17:07:45.889631 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:45.889479 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" podUID="8264e4fa-d20a-4d04-9fd2-384a1cc596b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 27 17:07:46.892824 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:46.892788 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:07:46.893273 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:46.893136 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:07:46.893935 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:46.893916 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d7q5z" Jul 27 17:07:47.598927 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:47.598895 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-vcgrx" Jul 27 17:07:47.599105 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:07:47.598989 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-bjtcc" Jul 27 17:08:11.267227 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:11.267191 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:08:26.736572 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.736533 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm"] Jul 27 17:08:26.739145 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.739125 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.743771 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.743749 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 27 17:08:26.743922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.743897 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 17:08:26.744738 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.744712 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6x9nq\"" Jul 27 17:08:26.744738 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.744732 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 17:08:26.756858 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.756833 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm"] Jul 27 17:08:26.796273 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.796245 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:08:26.824311 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.824285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b75df30-f5b2-4906-b499-42a88f0e1346-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.824461 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.824322 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.824461 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.824351 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.824461 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.824440 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.824596 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.824479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqb9f\" (UniqueName: \"kubernetes.io/projected/3b75df30-f5b2-4906-b499-42a88f0e1346-kube-api-access-lqb9f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.824596 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.824546 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.925952 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.925913 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.926114 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.926014 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b75df30-f5b2-4906-b499-42a88f0e1346-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.926114 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.926042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.926114 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.926080 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.926317 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.926120 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.926317 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.926229 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqb9f\" (UniqueName: \"kubernetes.io/projected/3b75df30-f5b2-4906-b499-42a88f0e1346-kube-api-access-lqb9f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.926438 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.926413 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.926507 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.926494 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.926642 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.926605 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.928421 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.928398 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b75df30-f5b2-4906-b499-42a88f0e1346-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.928513 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.928484 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b75df30-f5b2-4906-b499-42a88f0e1346-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:26.936456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:26.936436 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqb9f\" (UniqueName: \"kubernetes.io/projected/3b75df30-f5b2-4906-b499-42a88f0e1346-kube-api-access-lqb9f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vfsgm\" (UID: \"3b75df30-f5b2-4906-b499-42a88f0e1346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:27.050560 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:27.050479 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:27.191702 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:27.191661 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm"] Jul 27 17:08:27.195247 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:08:27.195223 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b75df30_f5b2_4906_b499_42a88f0e1346.slice/crio-1f62bfe3562074eb7ca4387bf07becab48e38e8d4dca79770706f23fada592bf WatchSource:0}: Error finding container 1f62bfe3562074eb7ca4387bf07becab48e38e8d4dca79770706f23fada592bf: Status 404 returned error can't find the container with id 1f62bfe3562074eb7ca4387bf07becab48e38e8d4dca79770706f23fada592bf Jul 27 17:08:27.819153 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:27.819113 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" event={"ID":"3b75df30-f5b2-4906-b499-42a88f0e1346","Type":"ContainerStarted","Data":"1f62bfe3562074eb7ca4387bf07becab48e38e8d4dca79770706f23fada592bf"} Jul 27 17:08:28.825070 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:28.825022 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" event={"ID":"3b75df30-f5b2-4906-b499-42a88f0e1346","Type":"ContainerStarted","Data":"505a6a503dacb23aeee89cfddf6f73d8ebed69e3bc189225bafbcbbba4f2cd3d"} Jul 27 17:08:28.847035 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:28.846977 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" podStartSLOduration=1.832726794 podStartE2EDuration="2.846957906s" totalImagesPullingTime="1.014231112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:28.845992445 +0000 UTC m=+705.316318634" watchObservedRunningTime="2026-07-27 17:08:28.846957906 +0000 UTC m=+705.317284101" Jul 27 17:08:37.051295 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:37.051205 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:37.051295 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:37.051279 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:37.063436 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:37.063410 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:37.879311 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:37.879281 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vfsgm" Jul 27 17:08:40.289810 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.289781 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd"] Jul 27 17:08:40.292502 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.292474 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.294931 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.294906 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 27 17:08:40.305250 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.305227 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd"] Jul 27 17:08:40.317066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.317036 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:08:40.441887 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.441853 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.441887 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.441887 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.442087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.441973 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2639b0f8-517e-4cfb-b3bb-6569c963ef17-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.442087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.442026 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzz9\" (UniqueName: \"kubernetes.io/projected/2639b0f8-517e-4cfb-b3bb-6569c963ef17-kube-api-access-wmzz9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.442087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.442051 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.442221 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.442098 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.542731 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.542653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2639b0f8-517e-4cfb-b3bb-6569c963ef17-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.542731 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.542689 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzz9\" (UniqueName: \"kubernetes.io/projected/2639b0f8-517e-4cfb-b3bb-6569c963ef17-kube-api-access-wmzz9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.542731 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.542710 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.542946 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.542832 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.542946 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.542907 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.542946 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.542933 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.543089 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.543070 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.543247 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.543229 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.543325 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.543289 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.545252 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.545228 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2639b0f8-517e-4cfb-b3bb-6569c963ef17-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.545365 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.545343 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2639b0f8-517e-4cfb-b3bb-6569c963ef17-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.550149 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.550124 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzz9\" (UniqueName: \"kubernetes.io/projected/2639b0f8-517e-4cfb-b3bb-6569c963ef17-kube-api-access-wmzz9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-trzsd\" (UID: \"2639b0f8-517e-4cfb-b3bb-6569c963ef17\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.602783 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.602758 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:40.729106 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.728971 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd"] Jul 27 17:08:40.731188 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:08:40.731151 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2639b0f8_517e_4cfb_b3bb_6569c963ef17.slice/crio-8d0d01bba6ece401fdc784db69f5d4f2dd10c44c9f1ffa04d2a6bc5663e2aba5 WatchSource:0}: Error finding container 8d0d01bba6ece401fdc784db69f5d4f2dd10c44c9f1ffa04d2a6bc5663e2aba5: Status 404 returned error can't find the container with id 8d0d01bba6ece401fdc784db69f5d4f2dd10c44c9f1ffa04d2a6bc5663e2aba5 Jul 27 17:08:40.880784 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:40.880750 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" event={"ID":"2639b0f8-517e-4cfb-b3bb-6569c963ef17","Type":"ContainerStarted","Data":"8d0d01bba6ece401fdc784db69f5d4f2dd10c44c9f1ffa04d2a6bc5663e2aba5"} Jul 27 17:08:41.886247 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:41.886208 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" event={"ID":"2639b0f8-517e-4cfb-b3bb-6569c963ef17","Type":"ContainerStarted","Data":"c18762fa6a7c3a753203354ca585a5d0c24ff847ea77c4ffb381e3f1cf93da91"} Jul 27 17:08:41.911373 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:41.911321 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" podStartSLOduration=1.6641593590000001 podStartE2EDuration="1.911304751s" totalImagesPullingTime="247.145392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:41.909840106 +0000 UTC m=+718.380166313" watchObservedRunningTime="2026-07-27 17:08:41.911304751 +0000 UTC m=+718.381630949" Jul 27 17:08:50.603511 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:50.603475 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:50.603962 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:50.603535 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:50.615841 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:50.615810 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:08:50.934696 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:08:50.934665 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-trzsd" Jul 27 17:09:00.073148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.073111 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2"] Jul 27 17:09:00.077616 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.077593 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.079742 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.079721 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 27 17:09:00.092484 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.092456 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2"] Jul 27 17:09:00.108884 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.108853 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwgt\" (UniqueName: \"kubernetes.io/projected/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-kube-api-access-smwgt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.109044 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.108896 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.109044 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.108923 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.109044 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.109005 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.109166 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.109044 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.109166 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.109129 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.119044 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.119018 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:09:00.210166 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210126 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.210166 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210166 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.210446 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210250 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.210446 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210272 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwgt\" (UniqueName: \"kubernetes.io/projected/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-kube-api-access-smwgt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.210446 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210295 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.210446 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.210662 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210632 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.210704 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210638 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.210704 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.210685 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.212686 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.212661 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.212775 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.212668 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.218420 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.218394 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwgt\" (UniqueName: \"kubernetes.io/projected/123ae25c-8068-40c2-b9eb-9c5e295bcc9d-kube-api-access-smwgt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2\" (UID: \"123ae25c-8068-40c2-b9eb-9c5e295bcc9d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.391168 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.391131 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:00.523651 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.523625 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2"] Jul 27 17:09:00.525778 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:09:00.525750 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123ae25c_8068_40c2_b9eb_9c5e295bcc9d.slice/crio-637f872bacefdf4d54c4a667cee7566e0b2a602457e8f197cc7c4cbd7e672260 WatchSource:0}: Error finding container 637f872bacefdf4d54c4a667cee7566e0b2a602457e8f197cc7c4cbd7e672260: Status 404 returned error can't find the container with id 637f872bacefdf4d54c4a667cee7566e0b2a602457e8f197cc7c4cbd7e672260 Jul 27 17:09:00.967538 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.967487 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" event={"ID":"123ae25c-8068-40c2-b9eb-9c5e295bcc9d","Type":"ContainerStarted","Data":"6438beaea28b2d3d6eb410c562f442f291a25fa354f6cb3490176f8fa1688dce"} Jul 27 17:09:00.967702 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.967586 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" event={"ID":"123ae25c-8068-40c2-b9eb-9c5e295bcc9d","Type":"ContainerStarted","Data":"637f872bacefdf4d54c4a667cee7566e0b2a602457e8f197cc7c4cbd7e672260"} Jul 27 17:09:00.992102 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:00.991960 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" podStartSLOduration=0.795952258 podStartE2EDuration="991.940147ms" totalImagesPullingTime="195.987889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:00.990832746 +0000 UTC m=+737.461158938" watchObservedRunningTime="2026-07-27 17:09:00.991940147 +0000 UTC m=+737.462266347" Jul 27 17:09:07.798805 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:07.798772 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:09:07.856967 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:07.856936 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:09:10.282249 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:10.282209 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:09:10.392127 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:10.392092 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:10.392303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:10.392244 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:10.404348 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:10.404327 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:10.550526 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:10.550452 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:09:11.022635 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:11.022608 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2" Jul 27 17:09:15.858534 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:09:15.858495 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:10:57.648434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:10:57.648392 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:10:57.773781 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:10:57.773746 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:10:58.444097 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:10:58.444067 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:10:58.718010 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:10:58.717921 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:10:58.837598 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:10:58.837565 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:11:23.422567 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:23.422534 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:11:23.522093 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:23.522062 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:11:44.756719 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:44.756684 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-s76xr"] Jul 27 17:11:44.757163 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:44.756873 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" podUID="5a758cfc-217f-4cf2-b494-19a57a5a81c5" containerName="manager" containerID="cri-o://c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6" gracePeriod=30 Jul 27 17:11:45.002950 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.002925 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:11:45.125978 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.125898 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert\") pod \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " Jul 27 17:11:45.126114 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.125994 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv6t5\" (UniqueName: \"kubernetes.io/projected/5a758cfc-217f-4cf2-b494-19a57a5a81c5-kube-api-access-xv6t5\") pod \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\" (UID: \"5a758cfc-217f-4cf2-b494-19a57a5a81c5\") " Jul 27 17:11:45.127926 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.127897 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a758cfc-217f-4cf2-b494-19a57a5a81c5-kube-api-access-xv6t5" (OuterVolumeSpecName: "kube-api-access-xv6t5") pod "5a758cfc-217f-4cf2-b494-19a57a5a81c5" (UID: "5a758cfc-217f-4cf2-b494-19a57a5a81c5"). InnerVolumeSpecName "kube-api-access-xv6t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:11:45.128040 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.127990 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5a758cfc-217f-4cf2-b494-19a57a5a81c5" (UID: "5a758cfc-217f-4cf2-b494-19a57a5a81c5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:11:45.226844 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.226813 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a758cfc-217f-4cf2-b494-19a57a5a81c5-webhook-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:11:45.226844 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.226839 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6t5\" (UniqueName: \"kubernetes.io/projected/5a758cfc-217f-4cf2-b494-19a57a5a81c5-kube-api-access-xv6t5\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:11:45.684765 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.684732 2557 generic.go:356] "Generic (PLEG): container finished" podID="5a758cfc-217f-4cf2-b494-19a57a5a81c5" containerID="c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6" exitCode=0 Jul 27 17:11:45.684936 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.684790 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" Jul 27 17:11:45.684936 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.684819 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" event={"ID":"5a758cfc-217f-4cf2-b494-19a57a5a81c5","Type":"ContainerDied","Data":"c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6"} Jul 27 17:11:45.684936 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.684891 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-s76xr" event={"ID":"5a758cfc-217f-4cf2-b494-19a57a5a81c5","Type":"ContainerDied","Data":"9e83188ac3553f67787e05c3dcd9e100efbfbb6d850a623ae7ff6861fd04ca7d"} Jul 27 17:11:45.684936 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.684913 2557 scope.go:117] "RemoveContainer" containerID="c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6" Jul 27 17:11:45.693200 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.693167 2557 scope.go:117] "RemoveContainer" containerID="c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6" Jul 27 17:11:45.693448 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:11:45.693428 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6\": container with ID starting with c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6 not found: ID does not exist" containerID="c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6" Jul 27 17:11:45.693496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.693456 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6"} err="failed to get container status \"c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6\": rpc error: code = NotFound desc = could not find container \"c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6\": container with ID starting with c8bac5e594a934d93db2704b611c10b7490e9ac464d7ff2b08e7794c00f5f4a6 not found: ID does not exist" Jul 27 17:11:45.707019 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.706993 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-s76xr"] Jul 27 17:11:45.709362 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:45.709340 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-s76xr"] Jul 27 17:11:46.064603 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:11:46.064526 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a758cfc-217f-4cf2-b494-19a57a5a81c5" path="/var/lib/kubelet/pods/5a758cfc-217f-4cf2-b494-19a57a5a81c5/volumes" Jul 27 17:12:46.630095 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.630061 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-648cc"] Jul 27 17:12:46.630533 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.630394 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a758cfc-217f-4cf2-b494-19a57a5a81c5" containerName="manager" Jul 27 17:12:46.630533 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.630405 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a758cfc-217f-4cf2-b494-19a57a5a81c5" containerName="manager" Jul 27 17:12:46.630533 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.630471 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a758cfc-217f-4cf2-b494-19a57a5a81c5" containerName="manager" Jul 27 17:12:46.632623 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.632601 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:46.635401 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.635381 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 17:12:46.635561 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.635424 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h2wn4\"" Jul 27 17:12:46.635688 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.635673 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 17:12:46.645244 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.645225 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-648cc"] Jul 27 17:12:46.746065 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.746035 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsmd\" (UniqueName: \"kubernetes.io/projected/583a3f03-d492-4026-8e0b-ac7c2c7456c4-kube-api-access-wmsmd\") pod \"maas-controller-74cbc56ff7-648cc\" (UID: \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\") " pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:46.746248 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.746085 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/583a3f03-d492-4026-8e0b-ac7c2c7456c4-webhook-cert\") pod \"maas-controller-74cbc56ff7-648cc\" (UID: \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\") " pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:46.847034 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.847003 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/583a3f03-d492-4026-8e0b-ac7c2c7456c4-webhook-cert\") pod \"maas-controller-74cbc56ff7-648cc\" (UID: \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\") " pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:46.847216 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.847083 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsmd\" (UniqueName: \"kubernetes.io/projected/583a3f03-d492-4026-8e0b-ac7c2c7456c4-kube-api-access-wmsmd\") pod \"maas-controller-74cbc56ff7-648cc\" (UID: \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\") " pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:46.849443 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.849419 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/583a3f03-d492-4026-8e0b-ac7c2c7456c4-webhook-cert\") pod \"maas-controller-74cbc56ff7-648cc\" (UID: \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\") " pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:46.857864 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.857840 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsmd\" (UniqueName: \"kubernetes.io/projected/583a3f03-d492-4026-8e0b-ac7c2c7456c4-kube-api-access-wmsmd\") pod \"maas-controller-74cbc56ff7-648cc\" (UID: \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\") " pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:46.942851 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:46.942769 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:47.071337 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:47.071074 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-648cc"] Jul 27 17:12:47.073520 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:12:47.073493 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583a3f03_d492_4026_8e0b_ac7c2c7456c4.slice/crio-69301fd40c34fb5ad61c6a4af40a61baad86b16bebb26e5acc6dfd737c792ca7 WatchSource:0}: Error finding container 69301fd40c34fb5ad61c6a4af40a61baad86b16bebb26e5acc6dfd737c792ca7: Status 404 returned error can't find the container with id 69301fd40c34fb5ad61c6a4af40a61baad86b16bebb26e5acc6dfd737c792ca7 Jul 27 17:12:47.074734 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:47.074718 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:12:47.958844 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:47.958755 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-648cc" event={"ID":"583a3f03-d492-4026-8e0b-ac7c2c7456c4","Type":"ContainerStarted","Data":"19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58"} Jul 27 17:12:47.958844 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:47.958811 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-648cc" event={"ID":"583a3f03-d492-4026-8e0b-ac7c2c7456c4","Type":"ContainerStarted","Data":"69301fd40c34fb5ad61c6a4af40a61baad86b16bebb26e5acc6dfd737c792ca7"} Jul 27 17:12:47.959274 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:47.958850 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:12:47.978336 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:47.978288 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74cbc56ff7-648cc" podStartSLOduration=1.420727791 podStartE2EDuration="1.978272387s" totalImagesPullingTime="557.544596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:12:47.976457477 +0000 UTC m=+964.446783677" watchObservedRunningTime="2026-07-27 17:12:47.978272387 +0000 UTC m=+964.448598581" Jul 27 17:12:58.966026 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:12:58.965990 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:13:03.718392 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:03.718352 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:04.026001 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.025913 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-648cc"] Jul 27 17:13:04.026240 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.026116 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74cbc56ff7-648cc" podUID="583a3f03-d492-4026-8e0b-ac7c2c7456c4" containerName="manager" containerID="cri-o://19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58" gracePeriod=30 Jul 27 17:13:04.267470 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.267446 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:13:04.304065 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.303986 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmsmd\" (UniqueName: \"kubernetes.io/projected/583a3f03-d492-4026-8e0b-ac7c2c7456c4-kube-api-access-wmsmd\") pod \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\" (UID: \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\") " Jul 27 17:13:04.304294 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.304086 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/583a3f03-d492-4026-8e0b-ac7c2c7456c4-webhook-cert\") pod \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\" (UID: \"583a3f03-d492-4026-8e0b-ac7c2c7456c4\") " Jul 27 17:13:04.306128 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.306101 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583a3f03-d492-4026-8e0b-ac7c2c7456c4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "583a3f03-d492-4026-8e0b-ac7c2c7456c4" (UID: "583a3f03-d492-4026-8e0b-ac7c2c7456c4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:13:04.306258 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.306242 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583a3f03-d492-4026-8e0b-ac7c2c7456c4-kube-api-access-wmsmd" (OuterVolumeSpecName: "kube-api-access-wmsmd") pod "583a3f03-d492-4026-8e0b-ac7c2c7456c4" (UID: "583a3f03-d492-4026-8e0b-ac7c2c7456c4"). InnerVolumeSpecName "kube-api-access-wmsmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:13:04.405233 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.405197 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/583a3f03-d492-4026-8e0b-ac7c2c7456c4-webhook-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:13:04.405233 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:04.405229 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmsmd\" (UniqueName: \"kubernetes.io/projected/583a3f03-d492-4026-8e0b-ac7c2c7456c4-kube-api-access-wmsmd\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:13:05.036497 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.036458 2557 generic.go:356] "Generic (PLEG): container finished" podID="583a3f03-d492-4026-8e0b-ac7c2c7456c4" containerID="19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58" exitCode=0 Jul 27 17:13:05.036892 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.036505 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-648cc" Jul 27 17:13:05.036892 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.036546 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-648cc" event={"ID":"583a3f03-d492-4026-8e0b-ac7c2c7456c4","Type":"ContainerDied","Data":"19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58"} Jul 27 17:13:05.036892 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.036600 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-648cc" event={"ID":"583a3f03-d492-4026-8e0b-ac7c2c7456c4","Type":"ContainerDied","Data":"69301fd40c34fb5ad61c6a4af40a61baad86b16bebb26e5acc6dfd737c792ca7"} Jul 27 17:13:05.036892 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.036629 2557 scope.go:117] "RemoveContainer" containerID="19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58" Jul 27 17:13:05.045527 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.045507 2557 scope.go:117] "RemoveContainer" containerID="19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58" Jul 27 17:13:05.045794 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:13:05.045775 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58\": container with ID starting with 19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58 not found: ID does not exist" containerID="19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58" Jul 27 17:13:05.045851 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.045803 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58"} err="failed to get container status \"19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58\": rpc error: code = NotFound desc = could not find container \"19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58\": container with ID starting with 19ffc22a85e47ce321f74236209d136066adb212debfa11aa82304855bc28b58 not found: ID does not exist" Jul 27 17:13:05.057091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.057065 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-648cc"] Jul 27 17:13:05.063785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:05.063759 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-648cc"] Jul 27 17:13:06.064446 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.064368 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="583a3f03-d492-4026-8e0b-ac7c2c7456c4" path="/var/lib/kubelet/pods/583a3f03-d492-4026-8e0b-ac7c2c7456c4/volumes" Jul 27 17:13:06.384514 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.384485 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:06.390509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.390477 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-2kjbn"] Jul 27 17:13:06.390835 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.390824 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583a3f03-d492-4026-8e0b-ac7c2c7456c4" containerName="manager" Jul 27 17:13:06.390889 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.390837 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="583a3f03-d492-4026-8e0b-ac7c2c7456c4" containerName="manager" Jul 27 17:13:06.390924 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.390894 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="583a3f03-d492-4026-8e0b-ac7c2c7456c4" containerName="manager" Jul 27 17:13:06.394484 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.394465 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:06.397245 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.397226 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 17:13:06.397364 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.397230 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h2wn4\"" Jul 27 17:13:06.397364 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.397264 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 17:13:06.405299 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.405277 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-2kjbn"] Jul 27 17:13:06.423635 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.423605 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd-webhook-cert\") pod \"maas-controller-74cbc56ff7-2kjbn\" (UID: \"f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd\") " pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:06.423809 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.423639 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjl5\" (UniqueName: \"kubernetes.io/projected/f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd-kube-api-access-ghjl5\") pod \"maas-controller-74cbc56ff7-2kjbn\" (UID: \"f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd\") " pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:06.524912 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.524874 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd-webhook-cert\") pod \"maas-controller-74cbc56ff7-2kjbn\" (UID: \"f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd\") " pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:06.524912 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.524911 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjl5\" (UniqueName: \"kubernetes.io/projected/f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd-kube-api-access-ghjl5\") pod \"maas-controller-74cbc56ff7-2kjbn\" (UID: \"f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd\") " pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:06.527304 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.527282 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd-webhook-cert\") pod \"maas-controller-74cbc56ff7-2kjbn\" (UID: \"f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd\") " pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:06.533605 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.533580 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjl5\" (UniqueName: \"kubernetes.io/projected/f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd-kube-api-access-ghjl5\") pod \"maas-controller-74cbc56ff7-2kjbn\" (UID: \"f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd\") " pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:06.705292 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.705198 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:06.834090 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:06.834063 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74cbc56ff7-2kjbn"] Jul 27 17:13:06.836154 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:13:06.836128 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dfe1a1_6e69_4cc4_bf01_98bf4a4a9dbd.slice/crio-e3da1f24aa09cd499d21d8c9ac25ae86d0dc6922e1a929cb0a1ff0037a3af3af WatchSource:0}: Error finding container e3da1f24aa09cd499d21d8c9ac25ae86d0dc6922e1a929cb0a1ff0037a3af3af: Status 404 returned error can't find the container with id e3da1f24aa09cd499d21d8c9ac25ae86d0dc6922e1a929cb0a1ff0037a3af3af Jul 27 17:13:07.047830 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:07.047736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" event={"ID":"f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd","Type":"ContainerStarted","Data":"e3da1f24aa09cd499d21d8c9ac25ae86d0dc6922e1a929cb0a1ff0037a3af3af"} Jul 27 17:13:08.053711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:08.053676 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" event={"ID":"f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd","Type":"ContainerStarted","Data":"923917d713c12a29c7000932677aed952fbb8db75e789f807df654d1963b9ff9"} Jul 27 17:13:08.054118 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:08.053799 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:08.078017 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:08.077971 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" podStartSLOduration=1.631101316 podStartE2EDuration="2.077955905s" totalImagesPullingTime="446.854589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:13:08.075656803 +0000 UTC m=+984.545983008" watchObservedRunningTime="2026-07-27 17:13:08.077955905 +0000 UTC m=+984.548282099" Jul 27 17:13:19.061380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:19.061349 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74cbc56ff7-2kjbn" Jul 27 17:13:42.471705 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:42.471669 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:42.606984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:42.606943 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:42.695236 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:42.695206 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:49.075845 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:49.075812 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:49.179029 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:49.178989 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:49.270823 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:49.270770 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:49.507971 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:49.507939 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:52.534330 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:52.534290 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:52.596692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:52.596657 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:55.478232 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:55.478201 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:13:55.543376 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:13:55.543342 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:14:55.161308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:14:55.161201 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:15:00.168970 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:00.168930 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn"] Jul 27 17:15:00.172102 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:00.172083 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" Jul 27 17:15:00.174608 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:00.174589 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grztm\"" Jul 27 17:15:00.181450 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:00.181434 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" Jul 27 17:15:00.226833 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:00.226802 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn"] Jul 27 17:15:00.318145 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:00.318122 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn"] Jul 27 17:15:00.320158 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:15:00.320128 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563d6016_d5ea_4b7f_8fb4_6b048c05047b.slice/crio-e37c43d77e70515f544fb18332385db4ecb02e181356a282bccede1e2bd2b4b6 WatchSource:0}: Error finding container e37c43d77e70515f544fb18332385db4ecb02e181356a282bccede1e2bd2b4b6: Status 404 returned error can't find the container with id e37c43d77e70515f544fb18332385db4ecb02e181356a282bccede1e2bd2b4b6 Jul 27 17:15:00.551475 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:00.551394 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" event={"ID":"563d6016-d5ea-4b7f-8fb4-6b048c05047b","Type":"ContainerStarted","Data":"e37c43d77e70515f544fb18332385db4ecb02e181356a282bccede1e2bd2b4b6"} Jul 27 17:15:03.569016 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:03.568904 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" event={"ID":"563d6016-d5ea-4b7f-8fb4-6b048c05047b","Type":"ContainerStarted","Data":"0d5478da4e43d5089324b59cdf44b0cdce5b14b89b6ea8523d19b007d8f35d97"} Jul 27 17:15:03.592126 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:03.592073 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" podStartSLOduration=1.066663701 podStartE2EDuration="3.59205736s" totalImagesPullingTime="2.525393659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:15:03.591310133 +0000 UTC m=+1100.061636329" watchObservedRunningTime="2026-07-27 17:15:03.59205736 +0000 UTC m=+1100.062383555" Jul 27 17:15:04.574850 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:04.574810 2557 generic.go:356] "Generic (PLEG): container finished" podID="563d6016-d5ea-4b7f-8fb4-6b048c05047b" containerID="0d5478da4e43d5089324b59cdf44b0cdce5b14b89b6ea8523d19b007d8f35d97" exitCode=0 Jul 27 17:15:04.575249 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:04.574897 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" event={"ID":"563d6016-d5ea-4b7f-8fb4-6b048c05047b","Type":"ContainerDied","Data":"0d5478da4e43d5089324b59cdf44b0cdce5b14b89b6ea8523d19b007d8f35d97"} Jul 27 17:15:05.711006 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:05.710983 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" Jul 27 17:15:06.585270 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:06.585243 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" Jul 27 17:15:06.585441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:06.585242 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752875-lr4bn" event={"ID":"563d6016-d5ea-4b7f-8fb4-6b048c05047b","Type":"ContainerDied","Data":"e37c43d77e70515f544fb18332385db4ecb02e181356a282bccede1e2bd2b4b6"} Jul 27 17:15:06.585441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:15:06.585366 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37c43d77e70515f544fb18332385db4ecb02e181356a282bccede1e2bd2b4b6" Jul 27 17:16:34.814054 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:16:34.813959 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:16:34.904996 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:16:34.904951 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:16:58.139162 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:16:58.139114 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:22:02.081373 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:22:02.081340 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:22:22.477752 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:22:22.477711 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:22:27.010353 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:22:27.010317 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:22:27.316557 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:22:27.316477 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:22:59.758971 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:22:59.758931 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:23:01.457508 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:23:01.457475 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:23:11.538015 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:23:11.537980 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:23:59.477025 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:23:59.476975 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:23:59.582043 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:23:59.582009 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:02.666887 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:02.666850 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:09.698218 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:09.698155 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:23.913378 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:23.913344 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:26.949480 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:26.949438 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:27.062197 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:27.062150 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:27.467093 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:27.467058 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:33.878548 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:33.878514 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:33.964213 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:33.964131 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:34.180596 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.180512 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:34.496771 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.496686 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq"] Jul 27 17:26:34.497135 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.497117 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563d6016-d5ea-4b7f-8fb4-6b048c05047b" containerName="cleanup" Jul 27 17:26:34.497263 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.497137 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="563d6016-d5ea-4b7f-8fb4-6b048c05047b" containerName="cleanup" Jul 27 17:26:34.497263 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.497252 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="563d6016-d5ea-4b7f-8fb4-6b048c05047b" containerName="cleanup" Jul 27 17:26:34.499252 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.499232 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.507039 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.506491 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-41d9ef37-openshift-default-dockercfg-99wx9\"" Jul 27 17:26:34.512856 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.512831 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:34.536621 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.536589 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq"] Jul 27 17:26:34.627318 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627281 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-certs\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.627504 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627333 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istiod-ca-cert\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.627504 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627396 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-socket\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.627504 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627414 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-envoy\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.627504 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627473 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-data\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.627504 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627494 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-token\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.627770 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627608 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-credential-socket\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.627770 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627651 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-podinfo\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.627770 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.627683 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48wq\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-kube-api-access-g48wq\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728200 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728136 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-credential-socket\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728399 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728207 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-podinfo\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728399 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728331 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48wq\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-kube-api-access-g48wq\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728519 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728404 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-certs\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728519 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istiod-ca-cert\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728519 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728472 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-socket\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728519 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728496 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-envoy\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728721 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728542 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-data\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728721 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728573 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-token\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728721 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728616 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-credential-socket\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.728881 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.728759 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-certs\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.729077 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.729046 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-socket\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.729077 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.729074 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-data\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.729295 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.729152 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istiod-ca-cert\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.730771 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.730752 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-podinfo\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.730858 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.730812 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-envoy\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.737407 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.737368 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-token\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.737588 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.737567 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48wq\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-kube-api-access-g48wq\") pod \"e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.810140 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.810054 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:34.860543 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.860319 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:34.967250 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.967217 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq"] Jul 27 17:26:34.978079 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:26:34.978044 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b92d74_e5bd_4872_b1f4_971c8f7fe54f.slice/crio-9b0221e8bfba7df6d94a321a9b114e81dbdc695ed9e1d9d225f8d0446b446d73 WatchSource:0}: Error finding container 9b0221e8bfba7df6d94a321a9b114e81dbdc695ed9e1d9d225f8d0446b446d73: Status 404 returned error can't find the container with id 9b0221e8bfba7df6d94a321a9b114e81dbdc695ed9e1d9d225f8d0446b446d73 Jul 27 17:26:34.980272 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.980234 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:26:34.980354 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.980309 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:26:34.980397 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:34.980353 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:26:35.599337 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:35.599243 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" event={"ID":"46b92d74-e5bd-4872-b1f4-971c8f7fe54f","Type":"ContainerStarted","Data":"55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa"} Jul 27 17:26:35.599337 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:35.599312 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" event={"ID":"46b92d74-e5bd-4872-b1f4-971c8f7fe54f","Type":"ContainerStarted","Data":"9b0221e8bfba7df6d94a321a9b114e81dbdc695ed9e1d9d225f8d0446b446d73"} Jul 27 17:26:35.626382 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:35.626327 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" podStartSLOduration=1.626309822 podStartE2EDuration="1.626309822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:26:35.624483835 +0000 UTC m=+1792.094810029" watchObservedRunningTime="2026-07-27 17:26:35.626309822 +0000 UTC m=+1792.096636016" Jul 27 17:26:35.810866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:35.810827 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:36.815592 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:36.815560 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:37.033304 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:37.033267 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:37.131048 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:37.131012 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:37.237403 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:37.237371 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:37.407054 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:37.406972 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:37.608244 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:37.608215 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:37.609047 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:37.609029 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:37.871662 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:37.871585 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:37.970241 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:37.970206 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:26:38.003264 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.003232 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p"] Jul 27 17:26:38.005628 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.005612 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.007638 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.007619 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-41d9ef37\"" Jul 27 17:26:38.017194 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.017151 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p"] Jul 27 17:26:38.158976 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.158934 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec26c2b6-2899-4e2b-bb76-d90977e23a96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p\" (UID: \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.159152 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.159076 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlxk9\" (UniqueName: \"kubernetes.io/projected/ec26c2b6-2899-4e2b-bb76-d90977e23a96-kube-api-access-nlxk9\") pod \"payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p\" (UID: \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.161860 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.161834 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn"] Jul 27 17:26:38.165016 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.164996 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:38.170017 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.169989 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-41d9ef37-dockercfg-7mxnc\"" Jul 27 17:26:38.181056 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.181024 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn"] Jul 27 17:26:38.260260 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.260223 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpkq\" (UniqueName: \"kubernetes.io/projected/663b7f6f-c34a-4040-b16d-8ea14c955a23-kube-api-access-lbpkq\") pod \"payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn\" (UID: \"663b7f6f-c34a-4040-b16d-8ea14c955a23\") " pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:38.260441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.260277 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec26c2b6-2899-4e2b-bb76-d90977e23a96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p\" (UID: \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.260441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.260385 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b7f6f-c34a-4040-b16d-8ea14c955a23-plugins-config-volume\") pod \"payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn\" (UID: \"663b7f6f-c34a-4040-b16d-8ea14c955a23\") " pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:38.260441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.260426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlxk9\" (UniqueName: \"kubernetes.io/projected/ec26c2b6-2899-4e2b-bb76-d90977e23a96-kube-api-access-nlxk9\") pod \"payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p\" (UID: \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.261050 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.261025 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec26c2b6-2899-4e2b-bb76-d90977e23a96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p\" (UID: \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.271439 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.271415 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlxk9\" (UniqueName: \"kubernetes.io/projected/ec26c2b6-2899-4e2b-bb76-d90977e23a96-kube-api-access-nlxk9\") pod \"payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p\" (UID: \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.317408 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.317377 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.360910 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.360874 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b7f6f-c34a-4040-b16d-8ea14c955a23-plugins-config-volume\") pod \"payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn\" (UID: \"663b7f6f-c34a-4040-b16d-8ea14c955a23\") " pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:38.361107 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.360946 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpkq\" (UniqueName: \"kubernetes.io/projected/663b7f6f-c34a-4040-b16d-8ea14c955a23-kube-api-access-lbpkq\") pod \"payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn\" (UID: \"663b7f6f-c34a-4040-b16d-8ea14c955a23\") " pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:38.361745 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.361723 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b7f6f-c34a-4040-b16d-8ea14c955a23-plugins-config-volume\") pod \"payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn\" (UID: \"663b7f6f-c34a-4040-b16d-8ea14c955a23\") " pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:38.371406 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.371377 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpkq\" (UniqueName: \"kubernetes.io/projected/663b7f6f-c34a-4040-b16d-8ea14c955a23-kube-api-access-lbpkq\") pod \"payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn\" (UID: \"663b7f6f-c34a-4040-b16d-8ea14c955a23\") " pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:38.462854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.462781 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p"] Jul 27 17:26:38.465750 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:26:38.465714 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec26c2b6_2899_4e2b_bb76_d90977e23a96.slice/crio-9091f14855f551444a50155bdc5bf041aebe4a9d3b94a440e90f14fd45ac3fb1 WatchSource:0}: Error finding container 9091f14855f551444a50155bdc5bf041aebe4a9d3b94a440e90f14fd45ac3fb1: Status 404 returned error can't find the container with id 9091f14855f551444a50155bdc5bf041aebe4a9d3b94a440e90f14fd45ac3fb1 Jul 27 17:26:38.479600 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.479575 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:38.616713 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.616679 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" event={"ID":"ec26c2b6-2899-4e2b-bb76-d90977e23a96","Type":"ContainerStarted","Data":"64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb"} Jul 27 17:26:38.616942 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.616743 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" event={"ID":"ec26c2b6-2899-4e2b-bb76-d90977e23a96","Type":"ContainerStarted","Data":"9091f14855f551444a50155bdc5bf041aebe4a9d3b94a440e90f14fd45ac3fb1"} Jul 27 17:26:38.617043 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.616997 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:38.618785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.618720 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn"] Jul 27 17:26:38.621007 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:26:38.620983 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663b7f6f_c34a_4040_b16d_8ea14c955a23.slice/crio-ceeb7f0c160c73e1fedad7693a09b4d49fdc0a323355e5c179da61ceef602729 WatchSource:0}: Error finding container ceeb7f0c160c73e1fedad7693a09b4d49fdc0a323355e5c179da61ceef602729: Status 404 returned error can't find the container with id ceeb7f0c160c73e1fedad7693a09b4d49fdc0a323355e5c179da61ceef602729 Jul 27 17:26:38.634916 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:38.634600 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" podStartSLOduration=1.6345820070000001 podStartE2EDuration="1.634582007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:26:38.634409943 +0000 UTC m=+1795.104736161" watchObservedRunningTime="2026-07-27 17:26:38.634582007 +0000 UTC m=+1795.104908202" Jul 27 17:26:39.622773 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:39.622724 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" event={"ID":"663b7f6f-c34a-4040-b16d-8ea14c955a23","Type":"ContainerStarted","Data":"e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b"} Jul 27 17:26:39.623234 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:39.622829 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" event={"ID":"663b7f6f-c34a-4040-b16d-8ea14c955a23","Type":"ContainerStarted","Data":"ceeb7f0c160c73e1fedad7693a09b4d49fdc0a323355e5c179da61ceef602729"} Jul 27 17:26:39.623234 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:39.623204 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:39.644511 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:39.644442 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" podStartSLOduration=1.64441885 podStartE2EDuration="1.64441885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:26:39.642292766 +0000 UTC m=+1796.112618959" watchObservedRunningTime="2026-07-27 17:26:39.64441885 +0000 UTC m=+1796.114745047" Jul 27 17:26:49.624876 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:49.624843 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:26:51.633666 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:51.633629 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:26:52.610391 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:52.610305 2557 patch_prober.go:28] interesting pod/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 27 17:26:52.610391 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:52.610367 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 27 17:26:52.690218 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:52.690162 2557 generic.go:356] "Generic (PLEG): container finished" podID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerID="55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa" exitCode=0 Jul 27 17:26:52.690598 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:52.690225 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" event={"ID":"46b92d74-e5bd-4872-b1f4-971c8f7fe54f","Type":"ContainerDied","Data":"55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa"} Jul 27 17:26:52.690643 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:52.690625 2557 scope.go:117] "RemoveContainer" containerID="55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa" Jul 27 17:26:52.691656 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:52.691625 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:26:52.691719 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:52.691695 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:26:52.691793 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:52.691730 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:26:53.696320 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:53.696285 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" event={"ID":"46b92d74-e5bd-4872-b1f4-971c8f7fe54f","Type":"ContainerStarted","Data":"f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93"} Jul 27 17:26:53.811236 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:53.811206 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:53.811442 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:53.811421 2557 patch_prober.go:28] interesting pod/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 27 17:26:53.811484 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:53.811467 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 27 17:26:54.811380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:54.811350 2557 patch_prober.go:28] interesting pod/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 27 17:26:54.811756 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:54.811402 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 27 17:26:55.810663 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:55.810627 2557 patch_prober.go:28] interesting pod/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 27 17:26:55.810833 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:55.810680 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 27 17:26:56.811244 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:56.811200 2557 patch_prober.go:28] interesting pod/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 27 17:26:56.811733 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:56.811270 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 27 17:26:57.811093 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:57.811047 2557 patch_prober.go:28] interesting pod/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 27 17:26:57.811298 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:57.811109 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 27 17:26:58.814883 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:58.814853 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:58.815271 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:58.815115 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:26:58.815721 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:26:58.815706 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:27:13.187081 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.187045 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:13.203345 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.203318 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn"] Jul 27 17:27:13.203603 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.203543 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" podUID="663b7f6f-c34a-4040-b16d-8ea14c955a23" containerName="payload-processing" containerID="cri-o://e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b" gracePeriod=30 Jul 27 17:27:13.215333 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.215306 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p"] Jul 27 17:27:13.215580 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.215540 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" podUID="ec26c2b6-2899-4e2b-bb76-d90977e23a96" containerName="payload-pre-processing" containerID="cri-o://64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb" gracePeriod=30 Jul 27 17:27:13.497942 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.497916 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:27:13.500833 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.500814 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:27:13.576883 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.576847 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec26c2b6-2899-4e2b-bb76-d90977e23a96-plugins-config-volume\") pod \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\" (UID: \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\") " Jul 27 17:27:13.577063 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.576898 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b7f6f-c34a-4040-b16d-8ea14c955a23-plugins-config-volume\") pod \"663b7f6f-c34a-4040-b16d-8ea14c955a23\" (UID: \"663b7f6f-c34a-4040-b16d-8ea14c955a23\") " Jul 27 17:27:13.577063 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.576943 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbpkq\" (UniqueName: \"kubernetes.io/projected/663b7f6f-c34a-4040-b16d-8ea14c955a23-kube-api-access-lbpkq\") pod \"663b7f6f-c34a-4040-b16d-8ea14c955a23\" (UID: \"663b7f6f-c34a-4040-b16d-8ea14c955a23\") " Jul 27 17:27:13.577063 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.576994 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlxk9\" (UniqueName: \"kubernetes.io/projected/ec26c2b6-2899-4e2b-bb76-d90977e23a96-kube-api-access-nlxk9\") pod \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\" (UID: \"ec26c2b6-2899-4e2b-bb76-d90977e23a96\") " Jul 27 17:27:13.577297 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.577274 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec26c2b6-2899-4e2b-bb76-d90977e23a96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec26c2b6-2899-4e2b-bb76-d90977e23a96" (UID: "ec26c2b6-2899-4e2b-bb76-d90977e23a96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:27:13.577350 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.577316 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663b7f6f-c34a-4040-b16d-8ea14c955a23-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "663b7f6f-c34a-4040-b16d-8ea14c955a23" (UID: "663b7f6f-c34a-4040-b16d-8ea14c955a23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:27:13.577410 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.577361 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec26c2b6-2899-4e2b-bb76-d90977e23a96-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:13.579211 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.579186 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec26c2b6-2899-4e2b-bb76-d90977e23a96-kube-api-access-nlxk9" (OuterVolumeSpecName: "kube-api-access-nlxk9") pod "ec26c2b6-2899-4e2b-bb76-d90977e23a96" (UID: "ec26c2b6-2899-4e2b-bb76-d90977e23a96"). InnerVolumeSpecName "kube-api-access-nlxk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:27:13.579313 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.579222 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663b7f6f-c34a-4040-b16d-8ea14c955a23-kube-api-access-lbpkq" (OuterVolumeSpecName: "kube-api-access-lbpkq") pod "663b7f6f-c34a-4040-b16d-8ea14c955a23" (UID: "663b7f6f-c34a-4040-b16d-8ea14c955a23"). InnerVolumeSpecName "kube-api-access-lbpkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:27:13.677961 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.677923 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/663b7f6f-c34a-4040-b16d-8ea14c955a23-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:13.677961 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.677953 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbpkq\" (UniqueName: \"kubernetes.io/projected/663b7f6f-c34a-4040-b16d-8ea14c955a23-kube-api-access-lbpkq\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:13.677961 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.677969 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlxk9\" (UniqueName: \"kubernetes.io/projected/ec26c2b6-2899-4e2b-bb76-d90977e23a96-kube-api-access-nlxk9\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:13.787369 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.787281 2557 generic.go:356] "Generic (PLEG): container finished" podID="ec26c2b6-2899-4e2b-bb76-d90977e23a96" containerID="64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb" exitCode=0 Jul 27 17:27:13.787369 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.787328 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" Jul 27 17:27:13.787369 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.787365 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" event={"ID":"ec26c2b6-2899-4e2b-bb76-d90977e23a96","Type":"ContainerDied","Data":"64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb"} Jul 27 17:27:13.787624 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.787415 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p" event={"ID":"ec26c2b6-2899-4e2b-bb76-d90977e23a96","Type":"ContainerDied","Data":"9091f14855f551444a50155bdc5bf041aebe4a9d3b94a440e90f14fd45ac3fb1"} Jul 27 17:27:13.787624 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.787439 2557 scope.go:117] "RemoveContainer" containerID="64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb" Jul 27 17:27:13.788752 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.788729 2557 generic.go:356] "Generic (PLEG): container finished" podID="663b7f6f-c34a-4040-b16d-8ea14c955a23" containerID="e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b" exitCode=0 Jul 27 17:27:13.788845 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.788763 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" Jul 27 17:27:13.788845 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.788759 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" event={"ID":"663b7f6f-c34a-4040-b16d-8ea14c955a23","Type":"ContainerDied","Data":"e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b"} Jul 27 17:27:13.788972 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.788876 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn" event={"ID":"663b7f6f-c34a-4040-b16d-8ea14c955a23","Type":"ContainerDied","Data":"ceeb7f0c160c73e1fedad7693a09b4d49fdc0a323355e5c179da61ceef602729"} Jul 27 17:27:13.796157 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.796141 2557 scope.go:117] "RemoveContainer" containerID="64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb" Jul 27 17:27:13.796487 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:27:13.796453 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb\": container with ID starting with 64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb not found: ID does not exist" containerID="64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb" Jul 27 17:27:13.796567 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.796481 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb"} err="failed to get container status \"64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb\": rpc error: code = NotFound desc = could not find container \"64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb\": container with ID starting with 64ba03d26721f8c1bc8e5959bc38ba5274c8336619d59e33a0922928951354bb not found: ID does not exist" Jul 27 17:27:13.796567 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.796506 2557 scope.go:117] "RemoveContainer" containerID="e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b" Jul 27 17:27:13.804089 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.804073 2557 scope.go:117] "RemoveContainer" containerID="e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b" Jul 27 17:27:13.804337 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:27:13.804319 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b\": container with ID starting with e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b not found: ID does not exist" containerID="e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b" Jul 27 17:27:13.804424 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.804344 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b"} err="failed to get container status \"e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b\": rpc error: code = NotFound desc = could not find container \"e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b\": container with ID starting with e192f7b73f1191a728b3c076daa7aeaea4e7d78e485edd43071e1a2c01e3ec4b not found: ID does not exist" Jul 27 17:27:13.813841 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.813820 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p"] Jul 27 17:27:13.821890 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.821870 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41d9ef37-6446d6cfdd-pml6p"] Jul 27 17:27:13.836491 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.836467 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn"] Jul 27 17:27:13.840699 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:13.840682 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-41d9ef37-ccbc9749d-vq4tn"] Jul 27 17:27:14.064618 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:14.064539 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="663b7f6f-c34a-4040-b16d-8ea14c955a23" path="/var/lib/kubelet/pods/663b7f6f-c34a-4040-b16d-8ea14c955a23/volumes" Jul 27 17:27:14.064956 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:14.064937 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec26c2b6-2899-4e2b-bb76-d90977e23a96" path="/var/lib/kubelet/pods/ec26c2b6-2899-4e2b-bb76-d90977e23a96/volumes" Jul 27 17:27:14.728714 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:14.728680 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:17.674875 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:17.674839 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:17.817719 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:17.817686 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq"] Jul 27 17:27:17.817937 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:17.817911 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" containerID="cri-o://f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93" gracePeriod=30 Jul 27 17:27:17.841934 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:17.841909 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:18.093124 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.093041 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:18.334388 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.334358 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:18.513169 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.513134 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:18.727531 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.727499 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:18.898259 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.898223 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw"] Jul 27 17:27:18.898672 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.898653 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec26c2b6-2899-4e2b-bb76-d90977e23a96" containerName="payload-pre-processing" Jul 27 17:27:18.898766 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.898675 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec26c2b6-2899-4e2b-bb76-d90977e23a96" containerName="payload-pre-processing" Jul 27 17:27:18.898766 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.898685 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663b7f6f-c34a-4040-b16d-8ea14c955a23" containerName="payload-processing" Jul 27 17:27:18.898766 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.898694 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="663b7f6f-c34a-4040-b16d-8ea14c955a23" containerName="payload-processing" Jul 27 17:27:18.898916 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.898789 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="663b7f6f-c34a-4040-b16d-8ea14c955a23" containerName="payload-processing" Jul 27 17:27:18.898916 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.898804 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec26c2b6-2899-4e2b-bb76-d90977e23a96" containerName="payload-pre-processing" Jul 27 17:27:18.900828 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.900807 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:18.903118 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.903098 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-26960311-gw-openshift-default-dockercfg-mr9rl\"" Jul 27 17:27:18.929112 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.929083 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw"] Jul 27 17:27:18.936308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:18.936280 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:19.021245 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021208 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-socket\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.021428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021293 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-data\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.021428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021322 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-token\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.021428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021342 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istiod-ca-cert\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.021428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021368 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvml7\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-kube-api-access-hvml7\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.021428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021410 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-envoy\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.021617 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021490 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-credential-socket\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.021617 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-certs\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.021617 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.021547 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-podinfo\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.042218 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.042170 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:19.122446 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122411 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-certs\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122446 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122447 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-podinfo\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122481 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-socket\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122558 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-data\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122580 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-token\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istiod-ca-cert\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122625 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvml7\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-kube-api-access-hvml7\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122679 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-envoy\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122716 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-credential-socket\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.122894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122876 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-certs\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.123049 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.122925 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-socket\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.123093 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.123046 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-credential-socket\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.123151 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.123124 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-data\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.123338 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.123317 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istiod-ca-cert\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.124890 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.124867 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-envoy\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.125079 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.125060 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-podinfo\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.131412 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.131378 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvml7\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-kube-api-access-hvml7\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.132651 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.132630 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-token\") pod \"e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.211899 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.211808 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:19.345026 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.345002 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw"] Jul 27 17:27:19.347785 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:27:19.347748 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657bd5f3_b2a4_4fc6_9b68_81adaa28896e.slice/crio-077a88ee6d2b145832232b28c64fdad4a92a3893c72236a9e6427478c8ac42ad WatchSource:0}: Error finding container 077a88ee6d2b145832232b28c64fdad4a92a3893c72236a9e6427478c8ac42ad: Status 404 returned error can't find the container with id 077a88ee6d2b145832232b28c64fdad4a92a3893c72236a9e6427478c8ac42ad Jul 27 17:27:19.350354 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.350317 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:27:19.350409 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.350387 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:27:19.350449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.350415 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:27:19.827977 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.827940 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" event={"ID":"657bd5f3-b2a4-4fc6-9b68-81adaa28896e","Type":"ContainerStarted","Data":"b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787"} Jul 27 17:27:19.828455 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.828006 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" event={"ID":"657bd5f3-b2a4-4fc6-9b68-81adaa28896e","Type":"ContainerStarted","Data":"077a88ee6d2b145832232b28c64fdad4a92a3893c72236a9e6427478c8ac42ad"} Jul 27 17:27:19.855496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:19.855446 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" podStartSLOduration=1.8554300160000001 podStartE2EDuration="1.855430016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:19.852226928 +0000 UTC m=+1836.322553124" watchObservedRunningTime="2026-07-27 17:27:19.855430016 +0000 UTC m=+1836.325756211" Jul 27 17:27:20.212380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:20.212345 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:20.217107 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:20.217085 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:20.832968 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:20.832927 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:20.833785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:20.833764 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:21.380307 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:21.380276 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:21.727209 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:21.727105 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:21.942564 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:21.942529 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:22.131503 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.131472 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:22.243963 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.243930 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck"] Jul 27 17:27:22.246653 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.246626 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:22.249714 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.249588 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-26960311\"" Jul 27 17:27:22.260247 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.260147 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck"] Jul 27 17:27:22.351157 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.351119 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwvt\" (UniqueName: \"kubernetes.io/projected/69a3fd73-6421-4154-8960-84c6e0311583-kube-api-access-bvwvt\") pod \"payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck\" (UID: \"69a3fd73-6421-4154-8960-84c6e0311583\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:22.351355 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.351269 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a3fd73-6421-4154-8960-84c6e0311583-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck\" (UID: \"69a3fd73-6421-4154-8960-84c6e0311583\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:22.452538 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.452450 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwvt\" (UniqueName: \"kubernetes.io/projected/69a3fd73-6421-4154-8960-84c6e0311583-kube-api-access-bvwvt\") pod \"payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck\" (UID: \"69a3fd73-6421-4154-8960-84c6e0311583\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:22.452689 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.452552 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a3fd73-6421-4154-8960-84c6e0311583-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck\" (UID: \"69a3fd73-6421-4154-8960-84c6e0311583\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:22.453277 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.453257 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a3fd73-6421-4154-8960-84c6e0311583-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck\" (UID: \"69a3fd73-6421-4154-8960-84c6e0311583\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:22.460747 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.460720 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwvt\" (UniqueName: \"kubernetes.io/projected/69a3fd73-6421-4154-8960-84c6e0311583-kube-api-access-bvwvt\") pod \"payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck\" (UID: \"69a3fd73-6421-4154-8960-84c6e0311583\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:22.560569 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.560538 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:22.919342 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.919319 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck"] Jul 27 17:27:22.921361 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:27:22.921331 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a3fd73_6421_4154_8960_84c6e0311583.slice/crio-173321b7faf619a8d54c337fb6542175e78c54c9cbdf7b414044d0d5afc18b9a WatchSource:0}: Error finding container 173321b7faf619a8d54c337fb6542175e78c54c9cbdf7b414044d0d5afc18b9a: Status 404 returned error can't find the container with id 173321b7faf619a8d54c337fb6542175e78c54c9cbdf7b414044d0d5afc18b9a Jul 27 17:27:22.953660 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:22.953630 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:23.051874 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.051839 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:27:23.160042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160011 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istiod-ca-cert\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160049 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g48wq\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-kube-api-access-g48wq\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160076 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-envoy\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160104 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-podinfo\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160128 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-data\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160143 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-credential-socket\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160224 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-certs\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160550 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160278 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-socket\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160550 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160340 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-token\") pod \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\" (UID: \"46b92d74-e5bd-4872-b1f4-971c8f7fe54f\") " Jul 27 17:27:23.160654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160548 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:27:23.160709 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160675 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.160761 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160721 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:27:23.160869 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160850 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:27:23.161016 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.160989 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:27:23.161016 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.161010 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-data" (OuterVolumeSpecName: "istio-data") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:27:23.162411 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.162381 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:27:23.162501 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.162411 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:27:23.162672 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.162649 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-kube-api-access-g48wq" (OuterVolumeSpecName: "kube-api-access-g48wq") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "kube-api-access-g48wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:27:23.162811 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.162795 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-token" (OuterVolumeSpecName: "istio-token") pod "46b92d74-e5bd-4872-b1f4-971c8f7fe54f" (UID: "46b92d74-e5bd-4872-b1f4-971c8f7fe54f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:27:23.261764 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.261731 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.261764 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.261759 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.261957 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.261773 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.261957 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.261786 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48wq\" (UniqueName: \"kubernetes.io/projected/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-kube-api-access-g48wq\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.261957 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.261798 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.261957 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.261808 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.261957 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.261818 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.261957 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.261831 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b92d74-e5bd-4872-b1f4-971c8f7fe54f-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:23.850117 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.850025 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" event={"ID":"69a3fd73-6421-4154-8960-84c6e0311583","Type":"ContainerStarted","Data":"657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374"} Jul 27 17:27:23.850117 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.850081 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" event={"ID":"69a3fd73-6421-4154-8960-84c6e0311583","Type":"ContainerStarted","Data":"173321b7faf619a8d54c337fb6542175e78c54c9cbdf7b414044d0d5afc18b9a"} Jul 27 17:27:23.850117 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.850109 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:23.856456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.852426 2557 generic.go:356] "Generic (PLEG): container finished" podID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerID="f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93" exitCode=0 Jul 27 17:27:23.856456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.852691 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" event={"ID":"46b92d74-e5bd-4872-b1f4-971c8f7fe54f","Type":"ContainerDied","Data":"f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93"} Jul 27 17:27:23.856456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.852790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" event={"ID":"46b92d74-e5bd-4872-b1f4-971c8f7fe54f","Type":"ContainerDied","Data":"9b0221e8bfba7df6d94a321a9b114e81dbdc695ed9e1d9d225f8d0446b446d73"} Jul 27 17:27:23.856456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.852877 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq" Jul 27 17:27:23.856456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.853068 2557 scope.go:117] "RemoveContainer" containerID="f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93" Jul 27 17:27:23.870612 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.870537 2557 scope.go:117] "RemoveContainer" containerID="55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa" Jul 27 17:27:23.872058 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.872024 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" podStartSLOduration=1.872010414 podStartE2EDuration="1.872010414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:23.87065465 +0000 UTC m=+1840.340980853" watchObservedRunningTime="2026-07-27 17:27:23.872010414 +0000 UTC m=+1840.342336610" Jul 27 17:27:23.879157 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.879139 2557 scope.go:117] "RemoveContainer" containerID="f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93" Jul 27 17:27:23.879444 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:27:23.879426 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93\": container with ID starting with f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93 not found: ID does not exist" containerID="f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93" Jul 27 17:27:23.879499 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.879453 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93"} err="failed to get container status \"f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93\": rpc error: code = NotFound desc = could not find container \"f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93\": container with ID starting with f36483f3e281f2e039195f9aa10910ee4e02a7e6ecb40f1dddf34415826b9d93 not found: ID does not exist" Jul 27 17:27:23.879499 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.879470 2557 scope.go:117] "RemoveContainer" containerID="55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa" Jul 27 17:27:23.879731 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:27:23.879713 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa\": container with ID starting with 55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa not found: ID does not exist" containerID="55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa" Jul 27 17:27:23.879789 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.879736 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa"} err="failed to get container status \"55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa\": rpc error: code = NotFound desc = could not find container \"55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa\": container with ID starting with 55c61235d2bece1977480561450437a0ee8cf7703435621998625ac1a6275caa not found: ID does not exist" Jul 27 17:27:23.885260 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.885239 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq"] Jul 27 17:27:23.891091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:23.891072 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-41d9ef37-openshift-default-578b6866d6-qv7rq"] Jul 27 17:27:24.065073 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:24.065035 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" path="/var/lib/kubelet/pods/46b92d74-e5bd-4872-b1f4-971c8f7fe54f/volumes" Jul 27 17:27:34.861793 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:34.861763 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:36.484687 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:36.484547 2557 patch_prober.go:28] interesting pod/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:44782->10.134.0.41:15021: read: connection reset by peer" start-of-body= Jul 27 17:27:36.484687 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:36.484615 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:44782->10.134.0.41:15021: read: connection reset by peer" Jul 27 17:27:36.917999 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:36.917967 2557 generic.go:356] "Generic (PLEG): container finished" podID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerID="b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787" exitCode=0 Jul 27 17:27:36.918203 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:36.918048 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" event={"ID":"657bd5f3-b2a4-4fc6-9b68-81adaa28896e","Type":"ContainerDied","Data":"b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787"} Jul 27 17:27:36.918479 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:36.918462 2557 scope.go:117] "RemoveContainer" containerID="b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787" Jul 27 17:27:36.919454 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:36.919425 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:27:36.919539 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:36.919515 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:27:36.919589 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:36.919548 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:27:37.924170 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:37.924135 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" event={"ID":"657bd5f3-b2a4-4fc6-9b68-81adaa28896e","Type":"ContainerStarted","Data":"510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa"} Jul 27 17:27:38.212247 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:38.212135 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:38.212408 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:38.212386 2557 patch_prober.go:28] interesting pod/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 27 17:27:38.212452 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:38.212433 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 27 17:27:39.212263 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:39.212226 2557 patch_prober.go:28] interesting pod/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 27 17:27:39.212652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:39.212292 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 27 17:27:40.213000 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:40.212963 2557 patch_prober.go:28] interesting pod/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 27 17:27:40.213428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:40.213029 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 27 17:27:41.212949 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:41.212912 2557 patch_prober.go:28] interesting pod/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 27 17:27:41.213136 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:41.212981 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 27 17:27:42.215341 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:42.215307 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:42.215743 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:42.215682 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:42.216379 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:42.216359 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:27:56.978511 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:56.978466 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:57.037669 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.037637 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck"] Jul 27 17:27:57.037948 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.037915 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" podUID="69a3fd73-6421-4154-8960-84c6e0311583" containerName="payload-pre-processing" containerID="cri-o://657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374" gracePeriod=30 Jul 27 17:27:57.074407 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.074380 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:57.184333 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.184295 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:27:57.300423 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.300396 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:57.388517 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.388486 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvwvt\" (UniqueName: \"kubernetes.io/projected/69a3fd73-6421-4154-8960-84c6e0311583-kube-api-access-bvwvt\") pod \"69a3fd73-6421-4154-8960-84c6e0311583\" (UID: \"69a3fd73-6421-4154-8960-84c6e0311583\") " Jul 27 17:27:57.388665 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.388534 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a3fd73-6421-4154-8960-84c6e0311583-plugins-config-volume\") pod \"69a3fd73-6421-4154-8960-84c6e0311583\" (UID: \"69a3fd73-6421-4154-8960-84c6e0311583\") " Jul 27 17:27:57.388943 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.388922 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a3fd73-6421-4154-8960-84c6e0311583-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69a3fd73-6421-4154-8960-84c6e0311583" (UID: "69a3fd73-6421-4154-8960-84c6e0311583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:27:57.390624 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.390599 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a3fd73-6421-4154-8960-84c6e0311583-kube-api-access-bvwvt" (OuterVolumeSpecName: "kube-api-access-bvwvt") pod "69a3fd73-6421-4154-8960-84c6e0311583" (UID: "69a3fd73-6421-4154-8960-84c6e0311583"). InnerVolumeSpecName "kube-api-access-bvwvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:27:57.489864 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.489834 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvwvt\" (UniqueName: \"kubernetes.io/projected/69a3fd73-6421-4154-8960-84c6e0311583-kube-api-access-bvwvt\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:57.489864 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:57.489861 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a3fd73-6421-4154-8960-84c6e0311583-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:27:58.015071 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.015038 2557 generic.go:356] "Generic (PLEG): container finished" podID="69a3fd73-6421-4154-8960-84c6e0311583" containerID="657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374" exitCode=0 Jul 27 17:27:58.015526 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.015077 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" Jul 27 17:27:58.015526 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.015092 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" event={"ID":"69a3fd73-6421-4154-8960-84c6e0311583","Type":"ContainerDied","Data":"657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374"} Jul 27 17:27:58.015526 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.015139 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck" event={"ID":"69a3fd73-6421-4154-8960-84c6e0311583","Type":"ContainerDied","Data":"173321b7faf619a8d54c337fb6542175e78c54c9cbdf7b414044d0d5afc18b9a"} Jul 27 17:27:58.015526 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.015161 2557 scope.go:117] "RemoveContainer" containerID="657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374" Jul 27 17:27:58.024836 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.024812 2557 scope.go:117] "RemoveContainer" containerID="657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374" Jul 27 17:27:58.025106 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:27:58.025087 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374\": container with ID starting with 657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374 not found: ID does not exist" containerID="657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374" Jul 27 17:27:58.025170 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.025115 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374"} err="failed to get container status \"657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374\": rpc error: code = NotFound desc = could not find container \"657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374\": container with ID starting with 657a00599c8427472c03547595db82892c5e3904641fb870c156d6c85b4c7374 not found: ID does not exist" Jul 27 17:27:58.037264 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.037241 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck"] Jul 27 17:27:58.042079 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.042058 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26960311-54cfc4cbcb-8bgck"] Jul 27 17:27:58.063588 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:27:58.063567 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a3fd73-6421-4154-8960-84c6e0311583" path="/var/lib/kubelet/pods/69a3fd73-6421-4154-8960-84c6e0311583/volumes" Jul 27 17:28:01.516256 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:01.516223 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:01.696836 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:01.696794 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw"] Jul 27 17:28:01.697110 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:01.697085 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" containerID="cri-o://510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa" gracePeriod=30 Jul 27 17:28:01.722565 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:01.722534 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:01.935674 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:01.935643 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:02.197049 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.196963 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:02.484007 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.483915 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv"] Jul 27 17:28:02.484519 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.484495 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" Jul 27 17:28:02.484519 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.484521 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" Jul 27 17:28:02.484717 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.484536 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" Jul 27 17:28:02.484717 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.484546 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" Jul 27 17:28:02.484717 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.484557 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a3fd73-6421-4154-8960-84c6e0311583" containerName="payload-pre-processing" Jul 27 17:28:02.484717 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.484566 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a3fd73-6421-4154-8960-84c6e0311583" containerName="payload-pre-processing" Jul 27 17:28:02.484717 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.484689 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" Jul 27 17:28:02.484717 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.484707 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a3fd73-6421-4154-8960-84c6e0311583" containerName="payload-pre-processing" Jul 27 17:28:02.487723 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.487700 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.505303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.505273 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv"] Jul 27 17:28:02.637434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637193 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-data\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.637434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637249 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-podinfo\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.637434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637281 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-certs\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.637434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637327 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-envoy\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.637434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637353 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istiod-ca-cert\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.637434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-socket\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.638129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637441 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-credential-socket\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.638129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637548 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn8h4\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-kube-api-access-hn8h4\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.638129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.637685 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-token\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738224 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738094 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-data\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738224 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738146 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-podinfo\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738458 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738382 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-certs\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738464 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-envoy\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738492 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istiod-ca-cert\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738628 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738523 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-socket\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738677 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738654 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-data\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738737 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738690 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-credential-socket\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738737 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738718 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-certs\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738825 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738747 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn8h4\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-kube-api-access-hn8h4\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.738873 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.738851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-token\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.739221 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.739198 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-socket\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.739320 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.739231 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-credential-socket\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.739702 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.739681 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istiod-ca-cert\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.740683 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.740660 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-envoy\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.740874 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.740856 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-podinfo\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.749378 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.749349 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn8h4\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-kube-api-access-hn8h4\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.749498 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.749455 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-token\") pod \"e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.751223 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.751199 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:02.799086 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.799050 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:02.952569 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.952539 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv"] Jul 27 17:28:02.953073 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:28:02.953045 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3a901b_ee28_4189_a5e0_180c4afdc5a6.slice/crio-ba9159a54849e9802742ed7f2775218180eff6fabdfd5ce15b6b7fb8c9b3edc7 WatchSource:0}: Error finding container ba9159a54849e9802742ed7f2775218180eff6fabdfd5ce15b6b7fb8c9b3edc7: Status 404 returned error can't find the container with id ba9159a54849e9802742ed7f2775218180eff6fabdfd5ce15b6b7fb8c9b3edc7 Jul 27 17:28:02.955233 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.955201 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:02.955307 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.955265 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:02.955363 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:02.955306 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:03.042830 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:03.042777 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" event={"ID":"bd3a901b-ee28-4189-a5e0-180c4afdc5a6","Type":"ContainerStarted","Data":"631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4"} Jul 27 17:28:03.043010 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:03.042846 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" event={"ID":"bd3a901b-ee28-4189-a5e0-180c4afdc5a6","Type":"ContainerStarted","Data":"ba9159a54849e9802742ed7f2775218180eff6fabdfd5ce15b6b7fb8c9b3edc7"} Jul 27 17:28:03.079380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:03.079317 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" podStartSLOduration=1.079293433 podStartE2EDuration="1.079293433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:03.078151404 +0000 UTC m=+1879.548477600" watchObservedRunningTime="2026-07-27 17:28:03.079293433 +0000 UTC m=+1879.549619630" Jul 27 17:28:03.122253 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:03.122201 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:03.799891 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:03.799856 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:04.804989 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:04.804953 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:05.006121 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:05.006089 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:05.054238 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:05.054204 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:05.055289 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:05.055218 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:05.106960 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:05.106915 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:05.216805 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:05.216752 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:05.637342 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:05.637256 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:05.873975 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:05.873940 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:05.971866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:05.971787 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:06.945650 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:06.945619 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:28:07.066169 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.066091 2557 generic.go:356] "Generic (PLEG): container finished" podID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerID="510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa" exitCode=0 Jul 27 17:28:07.066169 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.066133 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" Jul 27 17:28:07.066169 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.066138 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" event={"ID":"657bd5f3-b2a4-4fc6-9b68-81adaa28896e","Type":"ContainerDied","Data":"510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa"} Jul 27 17:28:07.066410 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.066214 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw" event={"ID":"657bd5f3-b2a4-4fc6-9b68-81adaa28896e","Type":"ContainerDied","Data":"077a88ee6d2b145832232b28c64fdad4a92a3893c72236a9e6427478c8ac42ad"} Jul 27 17:28:07.066410 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.066243 2557 scope.go:117] "RemoveContainer" containerID="510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa" Jul 27 17:28:07.074965 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.074948 2557 scope.go:117] "RemoveContainer" containerID="b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787" Jul 27 17:28:07.079125 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079096 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-credential-socket\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079229 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079151 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-podinfo\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079229 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079205 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-socket\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079356 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079246 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-envoy\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079356 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079283 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvml7\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-kube-api-access-hvml7\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079356 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079311 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-certs\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079356 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079339 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-token\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079559 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079399 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-data\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079559 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079404 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:07.079559 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079441 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istiod-ca-cert\") pod \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\" (UID: \"657bd5f3-b2a4-4fc6-9b68-81adaa28896e\") " Jul 27 17:28:07.079559 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079516 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:07.079770 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079745 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.079770 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079767 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.079884 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079797 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:07.079940 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.079921 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:28:07.081085 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.081054 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-data" (OuterVolumeSpecName: "istio-data") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:07.081688 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.081661 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:28:07.081906 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.081874 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-token" (OuterVolumeSpecName: "istio-token") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:28:07.081984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.081928 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:07.082089 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.082070 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-kube-api-access-hvml7" (OuterVolumeSpecName: "kube-api-access-hvml7") pod "657bd5f3-b2a4-4fc6-9b68-81adaa28896e" (UID: "657bd5f3-b2a4-4fc6-9b68-81adaa28896e"). InnerVolumeSpecName "kube-api-access-hvml7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:28:07.083742 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.083729 2557 scope.go:117] "RemoveContainer" containerID="510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa" Jul 27 17:28:07.084022 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:28:07.084001 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa\": container with ID starting with 510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa not found: ID does not exist" containerID="510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa" Jul 27 17:28:07.084104 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.084032 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa"} err="failed to get container status \"510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa\": rpc error: code = NotFound desc = could not find container \"510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa\": container with ID starting with 510c66a39587c419651c6e4dff824fcb97b809610425694a0e5a8105fd1025aa not found: ID does not exist" Jul 27 17:28:07.084104 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.084055 2557 scope.go:117] "RemoveContainer" containerID="b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787" Jul 27 17:28:07.084351 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:28:07.084333 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787\": container with ID starting with b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787 not found: ID does not exist" containerID="b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787" Jul 27 17:28:07.084400 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.084357 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787"} err="failed to get container status \"b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787\": rpc error: code = NotFound desc = could not find container \"b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787\": container with ID starting with b5295b0325836fe79f8f63fd8eb88a4d9d979a50eeb6114967cfdea4d03ac787 not found: ID does not exist" Jul 27 17:28:07.180818 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.180783 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.180818 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.180818 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.181070 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.180831 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvml7\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-kube-api-access-hvml7\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.181070 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.180844 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.181070 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.180858 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.181070 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.180870 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.181070 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.180882 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657bd5f3-b2a4-4fc6-9b68-81adaa28896e-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:07.404766 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.404736 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw"] Jul 27 17:28:07.412106 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:07.412083 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-26960311-gw-openshift-default-d4cfb854d-p4lrw"] Jul 27 17:28:08.065366 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:08.065329 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" path="/var/lib/kubelet/pods/657bd5f3-b2a4-4fc6-9b68-81adaa28896e/volumes" Jul 27 17:28:23.146918 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:23.146882 2557 generic.go:356] "Generic (PLEG): container finished" podID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerID="631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4" exitCode=0 Jul 27 17:28:23.147377 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:23.146937 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" event={"ID":"bd3a901b-ee28-4189-a5e0-180c4afdc5a6","Type":"ContainerDied","Data":"631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4"} Jul 27 17:28:23.147377 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:23.147308 2557 scope.go:117] "RemoveContainer" containerID="631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4" Jul 27 17:28:23.148232 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:23.148202 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:23.148297 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:23.148272 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:23.148338 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:23.148304 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:24.156430 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:24.156385 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" event={"ID":"bd3a901b-ee28-4189-a5e0-180c4afdc5a6","Type":"ContainerStarted","Data":"521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26"} Jul 27 17:28:24.799790 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:24.799759 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:24.799966 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:24.799951 2557 patch_prober.go:28] interesting pod/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 27 17:28:24.800017 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:24.799992 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 27 17:28:25.799471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:25.799433 2557 patch_prober.go:28] interesting pod/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 27 17:28:25.799842 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:25.799492 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 27 17:28:26.800395 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:26.800359 2557 patch_prober.go:28] interesting pod/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 27 17:28:26.800780 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:26.800414 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 27 17:28:27.800340 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:27.800305 2557 patch_prober.go:28] interesting pod/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 27 17:28:27.800517 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:27.800359 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 27 17:28:28.803772 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:28.803738 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:28.804196 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:28.803988 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:28.804720 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:28.804702 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:42.120362 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:42.120329 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:42.215536 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:42.215505 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:42.313910 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:42.313875 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:45.974967 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:45.974935 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:47.212265 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.212228 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv"] Jul 27 17:28:47.212736 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.212510 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" containerID="cri-o://521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26" gracePeriod=30 Jul 27 17:28:47.228164 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.228136 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:47.422661 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.422630 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:47.659235 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.659203 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:47.959386 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.959316 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp"] Jul 27 17:28:47.959686 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.959672 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" Jul 27 17:28:47.959729 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.959687 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" Jul 27 17:28:47.959729 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.959698 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" Jul 27 17:28:47.959729 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.959703 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" Jul 27 17:28:47.959825 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.959779 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b92d74-e5bd-4872-b1f4-971c8f7fe54f" containerName="istio-proxy" Jul 27 17:28:47.959825 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.959809 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" Jul 27 17:28:47.962636 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.962614 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:47.966976 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.966958 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-834e32af-openshift-default-dockercfg-6g77n\"" Jul 27 17:28:47.978471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.978449 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp"] Jul 27 17:28:47.982135 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:47.982113 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:48.048600 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.048567 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-certs\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.048785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.048631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-envoy\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.048785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.048706 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-podinfo\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.048785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.048744 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-token\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.048922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.048810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istiod-ca-cert\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.048922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.048874 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7rq\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-kube-api-access-8j7rq\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.048922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.048903 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-socket\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.049078 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.048929 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-credential-socket\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.049078 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.049031 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-data\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.104479 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.104449 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:48.150318 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150287 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-data\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150358 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-certs\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-envoy\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150465 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-podinfo\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150489 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-token\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150747 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150552 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istiod-ca-cert\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150747 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150718 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7rq\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-kube-api-access-8j7rq\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150857 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150792 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-socket\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150857 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150830 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-data\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.150857 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.150836 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-credential-socket\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.151260 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.151226 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-socket\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.151441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.151404 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-credential-socket\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.151929 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.151901 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-certs\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.152577 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.152551 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istiod-ca-cert\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.153736 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.153713 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-envoy\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.154051 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.154027 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-podinfo\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.160660 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.160633 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7rq\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-kube-api-access-8j7rq\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.161837 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.161815 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-token\") pod \"e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.273589 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.273508 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:48.419428 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.419394 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp"] Jul 27 17:28:48.420376 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:28:48.420351 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3524c03d_2858_4ba3_a0f1_be6eb95e2dee.slice/crio-486e1d52615dc565484ac5a5fe7f3c991f39fbd6c54a2dc6c329ae7d373e47a6 WatchSource:0}: Error finding container 486e1d52615dc565484ac5a5fe7f3c991f39fbd6c54a2dc6c329ae7d373e47a6: Status 404 returned error can't find the container with id 486e1d52615dc565484ac5a5fe7f3c991f39fbd6c54a2dc6c329ae7d373e47a6 Jul 27 17:28:48.422648 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.422612 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:48.422726 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.422688 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:48.422726 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:48.422719 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:28:49.281711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:49.281674 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" event={"ID":"3524c03d-2858-4ba3-a0f1-be6eb95e2dee","Type":"ContainerStarted","Data":"61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8"} Jul 27 17:28:49.282078 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:49.281731 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" event={"ID":"3524c03d-2858-4ba3-a0f1-be6eb95e2dee","Type":"ContainerStarted","Data":"486e1d52615dc565484ac5a5fe7f3c991f39fbd6c54a2dc6c329ae7d373e47a6"} Jul 27 17:28:49.306514 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:49.306466 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" podStartSLOduration=2.306449089 podStartE2EDuration="2.306449089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:49.305097365 +0000 UTC m=+1925.775423559" watchObservedRunningTime="2026-07-27 17:28:49.306449089 +0000 UTC m=+1925.776775286" Jul 27 17:28:50.274147 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:50.274109 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:50.278772 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:50.278749 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:50.292574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:50.292545 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:50.293460 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:50.293436 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:28:50.724844 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:50.724809 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:50.826791 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:50.826757 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:50.925278 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:50.925225 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:51.253492 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:51.253457 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:51.486782 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:51.486741 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:51.589197 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:51.589108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:52.480980 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.480919 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:52.593423 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593389 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-credential-socket\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593466 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-envoy\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593498 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-certs\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593513 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-data\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593537 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-podinfo\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593587 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istiod-ca-cert\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593603 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-socket\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593638 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn8h4\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-kube-api-access-hn8h4\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593666 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-token\") pod \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\" (UID: \"bd3a901b-ee28-4189-a5e0-180c4afdc5a6\") " Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593683 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:52.593854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593763 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:52.594398 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593894 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:52.594398 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593954 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:52.594398 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593974 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:52.594398 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.593976 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:28:52.594606 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.594577 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:52.595767 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.595742 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:28:52.596031 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.595984 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:28:52.596031 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.595999 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:28:52.596031 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.596011 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-kube-api-access-hn8h4" (OuterVolumeSpecName: "kube-api-access-hn8h4") pod "bd3a901b-ee28-4189-a5e0-180c4afdc5a6" (UID: "bd3a901b-ee28-4189-a5e0-180c4afdc5a6"). InnerVolumeSpecName "kube-api-access-hn8h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:28:52.694598 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.694568 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:52.694598 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.694593 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:52.694598 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.694603 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:52.694823 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.694611 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:52.694823 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.694619 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:52.694823 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.694628 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn8h4\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-kube-api-access-hn8h4\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:52.694823 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:52.694638 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3a901b-ee28-4189-a5e0-180c4afdc5a6-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:28:53.306521 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.306484 2557 generic.go:356] "Generic (PLEG): container finished" podID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerID="521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26" exitCode=0 Jul 27 17:28:53.306697 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.306543 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" Jul 27 17:28:53.306697 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.306544 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" event={"ID":"bd3a901b-ee28-4189-a5e0-180c4afdc5a6","Type":"ContainerDied","Data":"521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26"} Jul 27 17:28:53.306697 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.306665 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv" event={"ID":"bd3a901b-ee28-4189-a5e0-180c4afdc5a6","Type":"ContainerDied","Data":"ba9159a54849e9802742ed7f2775218180eff6fabdfd5ce15b6b7fb8c9b3edc7"} Jul 27 17:28:53.306697 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.306695 2557 scope.go:117] "RemoveContainer" containerID="521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26" Jul 27 17:28:53.315526 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.315509 2557 scope.go:117] "RemoveContainer" containerID="631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4" Jul 27 17:28:53.323164 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.323149 2557 scope.go:117] "RemoveContainer" containerID="521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26" Jul 27 17:28:53.323445 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:28:53.323424 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26\": container with ID starting with 521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26 not found: ID does not exist" containerID="521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26" Jul 27 17:28:53.323509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.323456 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26"} err="failed to get container status \"521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26\": rpc error: code = NotFound desc = could not find container \"521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26\": container with ID starting with 521e93a9789d70a551268bda59fe3865ecb1e289b4ef4dc32d236d5a915e2f26 not found: ID does not exist" Jul 27 17:28:53.323509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.323479 2557 scope.go:117] "RemoveContainer" containerID="631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4" Jul 27 17:28:53.323739 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:28:53.323718 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4\": container with ID starting with 631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4 not found: ID does not exist" containerID="631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4" Jul 27 17:28:53.323835 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.323745 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4"} err="failed to get container status \"631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4\": rpc error: code = NotFound desc = could not find container \"631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4\": container with ID starting with 631322afbe47762b28a4da6b52fd8519e89a388ee9e8b51e36866348658791c4 not found: ID does not exist" Jul 27 17:28:53.331387 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.331368 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv"] Jul 27 17:28:53.339756 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:53.339734 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eda34c7f-openshift-default-765fcd9bfd-mfpqv"] Jul 27 17:28:54.064327 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:28:54.064296 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" path="/var/lib/kubelet/pods/bd3a901b-ee28-4189-a5e0-180c4afdc5a6/volumes" Jul 27 17:29:02.853621 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:02.853578 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:02.953607 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:02.953571 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:05.705631 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:05.705602 2557 patch_prober.go:28] interesting pod/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:45050->10.134.0.44:15021: read: connection reset by peer" start-of-body= Jul 27 17:29:05.706045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:05.705687 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:45050->10.134.0.44:15021: read: connection reset by peer" Jul 27 17:29:06.386294 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:06.386263 2557 generic.go:356] "Generic (PLEG): container finished" podID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerID="61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8" exitCode=0 Jul 27 17:29:06.386500 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:06.386342 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" event={"ID":"3524c03d-2858-4ba3-a0f1-be6eb95e2dee","Type":"ContainerDied","Data":"61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8"} Jul 27 17:29:06.386816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:06.386800 2557 scope.go:117] "RemoveContainer" containerID="61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8" Jul 27 17:29:06.387836 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:06.387806 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:29:06.387918 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:06.387874 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:29:06.387918 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:06.387909 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:29:07.392768 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:07.392734 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" event={"ID":"3524c03d-2858-4ba3-a0f1-be6eb95e2dee","Type":"ContainerStarted","Data":"7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159"} Jul 27 17:29:08.274439 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:08.274398 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:29:08.274721 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:08.274695 2557 patch_prober.go:28] interesting pod/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 27 17:29:08.274787 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:08.274765 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 27 17:29:09.274803 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:09.274756 2557 patch_prober.go:28] interesting pod/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 27 17:29:09.275304 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:09.274834 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 27 17:29:10.274238 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:10.274194 2557 patch_prober.go:28] interesting pod/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 27 17:29:10.274423 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:10.274271 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 27 17:29:11.274364 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:11.274326 2557 patch_prober.go:28] interesting pod/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 27 17:29:11.274782 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:11.274387 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 27 17:29:12.278001 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:12.277974 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:29:12.278380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:12.278221 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:29:12.278825 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:12.278809 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:29:26.115125 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:26.115090 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:26.202918 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:26.202890 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:26.272500 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:26.272468 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:30.651844 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:30.651810 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:30.801843 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:30.801799 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp"] Jul 27 17:29:30.802085 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:30.802058 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" containerID="cri-o://7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159" gracePeriod=30 Jul 27 17:29:30.826952 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:30.826928 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:31.051851 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:31.051761 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:31.526728 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:31.526688 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:31.612667 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:31.612632 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:31.713104 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:31.713073 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:33.237573 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:33.237529 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:36.055361 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.055335 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:29:36.170306 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170282 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-token\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170490 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170349 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-podinfo\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170490 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170368 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-credential-socket\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170490 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170386 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-envoy\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170490 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170410 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istiod-ca-cert\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170490 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170442 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-socket\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170755 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170505 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j7rq\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-kube-api-access-8j7rq\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170755 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170530 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-data\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170755 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170561 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-certs\") pod \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\" (UID: \"3524c03d-2858-4ba3-a0f1-be6eb95e2dee\") " Jul 27 17:29:36.170911 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170748 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:29:36.170911 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.170844 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:29:36.171069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.171020 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:29:36.171215 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.171067 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:29:36.171710 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.171682 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-data" (OuterVolumeSpecName: "istio-data") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:29:36.172641 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.172607 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:29:36.172731 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.172643 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-token" (OuterVolumeSpecName: "istio-token") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:29:36.172731 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.172677 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:29:36.172810 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.172784 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-kube-api-access-8j7rq" (OuterVolumeSpecName: "kube-api-access-8j7rq") pod "3524c03d-2858-4ba3-a0f1-be6eb95e2dee" (UID: "3524c03d-2858-4ba3-a0f1-be6eb95e2dee"). InnerVolumeSpecName "kube-api-access-8j7rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:29:36.271452 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271422 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j7rq\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-kube-api-access-8j7rq\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.271452 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271447 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.271452 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271456 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.271638 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271464 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.271638 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271472 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.271638 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271479 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.271638 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271487 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.271638 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271495 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.271638 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.271502 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3524c03d-2858-4ba3-a0f1-be6eb95e2dee-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:29:36.532743 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.532658 2557 generic.go:356] "Generic (PLEG): container finished" podID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerID="7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159" exitCode=0 Jul 27 17:29:36.532743 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.532711 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" Jul 27 17:29:36.532917 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.532738 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" event={"ID":"3524c03d-2858-4ba3-a0f1-be6eb95e2dee","Type":"ContainerDied","Data":"7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159"} Jul 27 17:29:36.532917 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.532794 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp" event={"ID":"3524c03d-2858-4ba3-a0f1-be6eb95e2dee","Type":"ContainerDied","Data":"486e1d52615dc565484ac5a5fe7f3c991f39fbd6c54a2dc6c329ae7d373e47a6"} Jul 27 17:29:36.532917 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.532815 2557 scope.go:117] "RemoveContainer" containerID="7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159" Jul 27 17:29:36.541574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.541559 2557 scope.go:117] "RemoveContainer" containerID="61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8" Jul 27 17:29:36.550647 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.550628 2557 scope.go:117] "RemoveContainer" containerID="7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159" Jul 27 17:29:36.550952 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:29:36.550928 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159\": container with ID starting with 7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159 not found: ID does not exist" containerID="7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159" Jul 27 17:29:36.551069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.550959 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159"} err="failed to get container status \"7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159\": rpc error: code = NotFound desc = could not find container \"7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159\": container with ID starting with 7f2c37defc7ecd362c592cb295ccfa2e7405d8f233844deef189bffa03d69159 not found: ID does not exist" Jul 27 17:29:36.551069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.550978 2557 scope.go:117] "RemoveContainer" containerID="61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8" Jul 27 17:29:36.551388 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:29:36.551366 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8\": container with ID starting with 61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8 not found: ID does not exist" containerID="61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8" Jul 27 17:29:36.551451 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.551391 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8"} err="failed to get container status \"61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8\": rpc error: code = NotFound desc = could not find container \"61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8\": container with ID starting with 61eb9ffaf9106c9b5d61fbde69f8fea56bc04a8bfc210be1a798c4ee515a1cb8 not found: ID does not exist" Jul 27 17:29:36.560586 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.560566 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp"] Jul 27 17:29:36.571664 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:36.571644 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-834e32af-openshift-default-67d5fb88d-cq2tp"] Jul 27 17:29:38.063963 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:38.063929 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" path="/var/lib/kubelet/pods/3524c03d-2858-4ba3-a0f1-be6eb95e2dee/volumes" Jul 27 17:29:39.234853 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:39.234808 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:39.338857 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:39.338826 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:39.652152 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:39.652117 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:39.756679 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:39.756644 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:39.856692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:39.856662 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:39.960460 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:39.960356 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:40.436809 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:29:40.436771 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:00.143566 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143533 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb"] Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143870 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143882 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143894 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143900 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143910 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143916 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143925 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143930 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143991 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.143998 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3524c03d-2858-4ba3-a0f1-be6eb95e2dee" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.144006 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.144012 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3a901b-ee28-4189-a5e0-180c4afdc5a6" containerName="istio-proxy" Jul 27 17:30:00.144045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.144018 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="657bd5f3-b2a4-4fc6-9b68-81adaa28896e" containerName="istio-proxy" Jul 27 17:30:00.145872 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.145856 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" Jul 27 17:30:00.148890 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.148870 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grztm\"" Jul 27 17:30:00.154689 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.154665 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb"] Jul 27 17:30:00.156115 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.156096 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" Jul 27 17:30:00.284080 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.284056 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb"] Jul 27 17:30:00.286701 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:30:00.286674 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef23af4_0e30_4bea_850c_3b25673d1e5c.slice/crio-7994b8968561af102a6317284d939832e8e6e2cd8df5f763f6e69b7f588cc05b WatchSource:0}: Error finding container 7994b8968561af102a6317284d939832e8e6e2cd8df5f763f6e69b7f588cc05b: Status 404 returned error can't find the container with id 7994b8968561af102a6317284d939832e8e6e2cd8df5f763f6e69b7f588cc05b Jul 27 17:30:00.638291 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.638253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" event={"ID":"aef23af4-0e30-4bea-850c-3b25673d1e5c","Type":"ContainerStarted","Data":"003d776cfdba3de72bdb3d4b53d74ff02e594ae26b3d6ac9ee83441dd7e7c2c2"} Jul 27 17:30:00.638488 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.638314 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" event={"ID":"aef23af4-0e30-4bea-850c-3b25673d1e5c","Type":"ContainerStarted","Data":"7994b8968561af102a6317284d939832e8e6e2cd8df5f763f6e69b7f588cc05b"} Jul 27 17:30:00.652756 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:00.652702 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" podStartSLOduration=0.652682714 podStartE2EDuration="652.682714ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:00.65246302 +0000 UTC m=+1997.122789212" watchObservedRunningTime="2026-07-27 17:30:00.652682714 +0000 UTC m=+1997.123008910" Jul 27 17:30:01.643615 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:01.643579 2557 generic.go:356] "Generic (PLEG): container finished" podID="aef23af4-0e30-4bea-850c-3b25673d1e5c" containerID="003d776cfdba3de72bdb3d4b53d74ff02e594ae26b3d6ac9ee83441dd7e7c2c2" exitCode=0 Jul 27 17:30:01.644099 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:01.643661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" event={"ID":"aef23af4-0e30-4bea-850c-3b25673d1e5c","Type":"ContainerDied","Data":"003d776cfdba3de72bdb3d4b53d74ff02e594ae26b3d6ac9ee83441dd7e7c2c2"} Jul 27 17:30:02.775669 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:02.775647 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" Jul 27 17:30:03.654300 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:03.654267 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" Jul 27 17:30:03.654509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:03.654297 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752890-557qb" event={"ID":"aef23af4-0e30-4bea-850c-3b25673d1e5c","Type":"ContainerDied","Data":"7994b8968561af102a6317284d939832e8e6e2cd8df5f763f6e69b7f588cc05b"} Jul 27 17:30:03.654509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:03.654374 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7994b8968561af102a6317284d939832e8e6e2cd8df5f763f6e69b7f588cc05b" Jul 27 17:30:06.368373 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:06.368328 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:12.682482 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:12.682440 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:12.785837 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:12.785799 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:12.882898 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:12.882847 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:28.795618 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:28.795582 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:34.764923 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:34.764888 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:34.867564 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:34.867531 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:37.530110 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:37.530079 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:45.492605 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:45.492561 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:46.270544 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:46.270509 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:53.752332 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:53.752292 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:53.824098 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:53.824072 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:53.907628 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:53.907595 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:54.008943 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:54.008866 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:54.109996 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:54.109962 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:54.827317 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:54.827280 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:54.925845 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:54.925819 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:55.026239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:55.026209 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:56.658071 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:30:56.658041 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:04.142904 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:04.142868 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:04.242090 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:04.242059 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:04.852426 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:04.852383 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:10.702232 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:10.702153 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:10.803276 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:10.803238 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:10.984541 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:10.984456 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:11.377615 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.377593 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq"] Jul 27 17:31:11.377940 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.377929 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef23af4-0e30-4bea-850c-3b25673d1e5c" containerName="cleanup" Jul 27 17:31:11.377983 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.377942 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef23af4-0e30-4bea-850c-3b25673d1e5c" containerName="cleanup" Jul 27 17:31:11.378021 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.378011 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef23af4-0e30-4bea-850c-3b25673d1e5c" containerName="cleanup" Jul 27 17:31:11.381032 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.380977 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.383760 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.383739 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-18117344-openshift-default-dockercfg-tts2m\"" Jul 27 17:31:11.398699 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.398677 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq"] Jul 27 17:31:11.439385 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.439343 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:11.464390 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464348 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-data\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.464546 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464409 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-envoy\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.464546 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464438 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f086aaa-1e01-4e56-81dc-7c4789303778-istiod-ca-cert\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.464546 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464469 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rgk\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-kube-api-access-d6rgk\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.464689 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464568 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-socket\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.464689 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464605 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-credential-socket\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.464689 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464629 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-token\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.464689 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464661 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-podinfo\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.464841 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.464685 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-certs\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566113 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566075 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-data\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566123 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-envoy\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566151 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f086aaa-1e01-4e56-81dc-7c4789303778-istiod-ca-cert\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566207 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rgk\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-kube-api-access-d6rgk\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566248 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-socket\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566270 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-credential-socket\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566290 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-token\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566539 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566323 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-podinfo\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566539 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566351 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-certs\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.566620 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.566536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-data\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.567092 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.567055 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f086aaa-1e01-4e56-81dc-7c4789303778-istiod-ca-cert\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.567092 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.567073 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-credential-socket\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.567092 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.567066 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-certs\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.567316 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.567115 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-socket\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.568844 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.568824 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-podinfo\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.568953 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.568845 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-envoy\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.578396 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.578364 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rgk\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-kube-api-access-d6rgk\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.582375 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.582352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-token\") pod \"e2e-mt-18117344-openshift-default-7667c5d984-87qbq\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.691683 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.691599 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:11.756593 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.756552 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:11.844555 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.844523 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq"] Jul 27 17:31:11.846075 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:31:11.846050 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f086aaa_1e01_4e56_81dc_7c4789303778.slice/crio-cb516c515025bcf7389a44c9ad0665801c55f823f67769b174dacddbf5dab56c WatchSource:0}: Error finding container cb516c515025bcf7389a44c9ad0665801c55f823f67769b174dacddbf5dab56c: Status 404 returned error can't find the container with id cb516c515025bcf7389a44c9ad0665801c55f823f67769b174dacddbf5dab56c Jul 27 17:31:11.848217 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.848164 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:31:11.848316 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.848265 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:31:11.848316 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.848310 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:31:11.957774 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.957681 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" event={"ID":"0f086aaa-1e01-4e56-81dc-7c4789303778","Type":"ContainerStarted","Data":"bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d"} Jul 27 17:31:11.957774 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.957738 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" event={"ID":"0f086aaa-1e01-4e56-81dc-7c4789303778","Type":"ContainerStarted","Data":"cb516c515025bcf7389a44c9ad0665801c55f823f67769b174dacddbf5dab56c"} Jul 27 17:31:11.990193 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:11.990127 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" podStartSLOduration=0.99010575 podStartE2EDuration="990.10575ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:11.987300166 +0000 UTC m=+2068.457626362" watchObservedRunningTime="2026-07-27 17:31:11.99010575 +0000 UTC m=+2068.460431954" Jul 27 17:31:12.692434 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:12.692404 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:12.697305 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:12.697279 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:12.963381 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:12.963300 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:12.964366 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:12.964345 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:14.286756 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:14.286713 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:14.372353 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:14.372318 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:14.472149 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:14.472120 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:14.620193 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:14.620109 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:15.367693 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.367663 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng"] Jul 27 17:31:15.371035 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.371018 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:15.373599 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.373577 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-18117344-dockercfg-66wrl\"" Jul 27 17:31:15.373710 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.373667 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-18117344\"" Jul 27 17:31:15.397074 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.397050 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng"] Jul 27 17:31:15.501075 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.501038 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmqb\" (UniqueName: \"kubernetes.io/projected/1ca3a202-c921-4464-be88-fc8316f32a07-kube-api-access-mwmqb\") pod \"payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng\" (UID: \"1ca3a202-c921-4464-be88-fc8316f32a07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:15.501286 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.501146 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3a202-c921-4464-be88-fc8316f32a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng\" (UID: \"1ca3a202-c921-4464-be88-fc8316f32a07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:15.602486 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.602447 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3a202-c921-4464-be88-fc8316f32a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng\" (UID: \"1ca3a202-c921-4464-be88-fc8316f32a07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:15.602634 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.602533 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmqb\" (UniqueName: \"kubernetes.io/projected/1ca3a202-c921-4464-be88-fc8316f32a07-kube-api-access-mwmqb\") pod \"payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng\" (UID: \"1ca3a202-c921-4464-be88-fc8316f32a07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:15.603114 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.603095 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3a202-c921-4464-be88-fc8316f32a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng\" (UID: \"1ca3a202-c921-4464-be88-fc8316f32a07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:15.613802 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.613774 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmqb\" (UniqueName: \"kubernetes.io/projected/1ca3a202-c921-4464-be88-fc8316f32a07-kube-api-access-mwmqb\") pod \"payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng\" (UID: \"1ca3a202-c921-4464-be88-fc8316f32a07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:15.682478 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.682451 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:15.820120 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.820090 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng"] Jul 27 17:31:15.823654 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:31:15.823619 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca3a202_c921_4464_be88_fc8316f32a07.slice/crio-ded4455f2ec9df1491609484a436a22b88bd22c2b45f17b51b3ff41193c38201 WatchSource:0}: Error finding container ded4455f2ec9df1491609484a436a22b88bd22c2b45f17b51b3ff41193c38201: Status 404 returned error can't find the container with id ded4455f2ec9df1491609484a436a22b88bd22c2b45f17b51b3ff41193c38201 Jul 27 17:31:15.979701 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.979622 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" event={"ID":"1ca3a202-c921-4464-be88-fc8316f32a07","Type":"ContainerStarted","Data":"bca98226e966e9a76458009c55df68fc97824998e69c6242b015bf59abf18c8c"} Jul 27 17:31:15.979701 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.979667 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" event={"ID":"1ca3a202-c921-4464-be88-fc8316f32a07","Type":"ContainerStarted","Data":"ded4455f2ec9df1491609484a436a22b88bd22c2b45f17b51b3ff41193c38201"} Jul 27 17:31:15.979917 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:15.979754 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:16.000094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:16.000049 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" podStartSLOduration=1.000035041 podStartE2EDuration="1.000035041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:15.998460435 +0000 UTC m=+2072.468786631" watchObservedRunningTime="2026-07-27 17:31:16.000035041 +0000 UTC m=+2072.470361257" Jul 27 17:31:17.215004 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:17.214970 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:17.304490 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:17.304453 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:18.687993 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:18.687958 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:19.375320 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:19.375284 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:19.467220 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:19.467169 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:19.803355 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:19.803325 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:25.893119 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:25.893084 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:25.997355 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:25.997325 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:26.027104 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:26.027078 2557 generic.go:356] "Generic (PLEG): container finished" podID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerID="bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d" exitCode=0 Jul 27 17:31:26.027291 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:26.027120 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" event={"ID":"0f086aaa-1e01-4e56-81dc-7c4789303778","Type":"ContainerDied","Data":"bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d"} Jul 27 17:31:26.027560 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:26.027542 2557 scope.go:117] "RemoveContainer" containerID="bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d" Jul 27 17:31:26.028590 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:26.028563 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:31:26.028647 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:26.028629 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:31:26.028683 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:26.028671 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:31:26.987164 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:26.987132 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:27.035128 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:27.035098 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" event={"ID":"0f086aaa-1e01-4e56-81dc-7c4789303778","Type":"ContainerStarted","Data":"09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977"} Jul 27 17:31:27.691987 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:27.691950 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:27.693091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:27.693057 2557 patch_prober.go:28] interesting pod/e2e-mt-18117344-openshift-default-7667c5d984-87qbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 27 17:31:27.693242 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:27.693113 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 27 17:31:27.963705 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:27.963608 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:28.692911 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:28.692875 2557 patch_prober.go:28] interesting pod/e2e-mt-18117344-openshift-default-7667c5d984-87qbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 27 17:31:28.693338 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:28.692927 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 27 17:31:29.692987 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:29.692940 2557 patch_prober.go:28] interesting pod/e2e-mt-18117344-openshift-default-7667c5d984-87qbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 27 17:31:29.693400 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:29.693008 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 27 17:31:30.692662 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:30.692628 2557 patch_prober.go:28] interesting pod/e2e-mt-18117344-openshift-default-7667c5d984-87qbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 27 17:31:30.692840 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:30.692682 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 27 17:31:31.695994 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:31.695964 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:31.696902 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:31.696882 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:36.855528 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:36.855493 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:36.911682 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:36.911652 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng"] Jul 27 17:31:36.911892 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:36.911870 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" podUID="1ca3a202-c921-4464-be88-fc8316f32a07" containerName="payload-pre-processing" containerID="cri-o://bca98226e966e9a76458009c55df68fc97824998e69c6242b015bf59abf18c8c" gracePeriod=30 Jul 27 17:31:36.957593 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:36.957564 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:36.985509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:36.985474 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" podUID="1ca3a202-c921-4464-be88-fc8316f32a07" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.47:9004: connect: connection refused" Jul 27 17:31:37.057526 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.057485 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:37.087795 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.087759 2557 generic.go:356] "Generic (PLEG): container finished" podID="1ca3a202-c921-4464-be88-fc8316f32a07" containerID="bca98226e966e9a76458009c55df68fc97824998e69c6242b015bf59abf18c8c" exitCode=0 Jul 27 17:31:37.087934 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.087831 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" event={"ID":"1ca3a202-c921-4464-be88-fc8316f32a07","Type":"ContainerDied","Data":"bca98226e966e9a76458009c55df68fc97824998e69c6242b015bf59abf18c8c"} Jul 27 17:31:37.158125 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.158103 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:37.302160 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.302123 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwmqb\" (UniqueName: \"kubernetes.io/projected/1ca3a202-c921-4464-be88-fc8316f32a07-kube-api-access-mwmqb\") pod \"1ca3a202-c921-4464-be88-fc8316f32a07\" (UID: \"1ca3a202-c921-4464-be88-fc8316f32a07\") " Jul 27 17:31:37.302367 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.302193 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3a202-c921-4464-be88-fc8316f32a07-plugins-config-volume\") pod \"1ca3a202-c921-4464-be88-fc8316f32a07\" (UID: \"1ca3a202-c921-4464-be88-fc8316f32a07\") " Jul 27 17:31:37.302541 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.302518 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca3a202-c921-4464-be88-fc8316f32a07-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ca3a202-c921-4464-be88-fc8316f32a07" (UID: "1ca3a202-c921-4464-be88-fc8316f32a07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:31:37.304211 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.304188 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca3a202-c921-4464-be88-fc8316f32a07-kube-api-access-mwmqb" (OuterVolumeSpecName: "kube-api-access-mwmqb") pod "1ca3a202-c921-4464-be88-fc8316f32a07" (UID: "1ca3a202-c921-4464-be88-fc8316f32a07"). InnerVolumeSpecName "kube-api-access-mwmqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:31:37.403646 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.403579 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca3a202-c921-4464-be88-fc8316f32a07-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:37.403646 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:37.403603 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmqb\" (UniqueName: \"kubernetes.io/projected/1ca3a202-c921-4464-be88-fc8316f32a07-kube-api-access-mwmqb\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:38.093580 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:38.093555 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" Jul 27 17:31:38.093945 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:38.093570 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng" event={"ID":"1ca3a202-c921-4464-be88-fc8316f32a07","Type":"ContainerDied","Data":"ded4455f2ec9df1491609484a436a22b88bd22c2b45f17b51b3ff41193c38201"} Jul 27 17:31:38.093945 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:38.093637 2557 scope.go:117] "RemoveContainer" containerID="bca98226e966e9a76458009c55df68fc97824998e69c6242b015bf59abf18c8c" Jul 27 17:31:38.113381 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:38.113358 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng"] Jul 27 17:31:38.118017 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:38.117994 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18117344-76dd5f4f56-wgjng"] Jul 27 17:31:40.064347 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:40.064314 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca3a202-c921-4464-be88-fc8316f32a07" path="/var/lib/kubelet/pods/1ca3a202-c921-4464-be88-fc8316f32a07/volumes" Jul 27 17:31:41.394385 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:41.394320 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:41.878068 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:41.878037 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:48.102361 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.102327 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:48.203251 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.203222 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:48.353202 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.353109 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq"] Jul 27 17:31:48.353398 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.353352 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" containerID="cri-o://09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977" gracePeriod=30 Jul 27 17:31:48.379546 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.379487 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:48.588993 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.588948 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:48.721023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.720984 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:48.814783 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.814739 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:48.927691 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:48.927660 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:53.595571 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.595545 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:53.639530 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639502 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-envoy\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.639674 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639533 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6rgk\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-kube-api-access-d6rgk\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.639674 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639608 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f086aaa-1e01-4e56-81dc-7c4789303778-istiod-ca-cert\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.639674 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639650 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-socket\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.639866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639698 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-token\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.639866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639718 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-credential-socket\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.639866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639738 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-podinfo\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.639866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639766 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-certs\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.639866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639825 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-data\") pod \"0f086aaa-1e01-4e56-81dc-7c4789303778\" (UID: \"0f086aaa-1e01-4e56-81dc-7c4789303778\") " Jul 27 17:31:53.640130 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.639971 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f086aaa-1e01-4e56-81dc-7c4789303778-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:31:53.640492 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.640211 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:31:53.640492 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.640230 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f086aaa-1e01-4e56-81dc-7c4789303778-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:53.640492 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.640439 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:31:53.640718 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.640526 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:31:53.640911 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.640885 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:31:53.641832 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.641799 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-kube-api-access-d6rgk" (OuterVolumeSpecName: "kube-api-access-d6rgk") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "kube-api-access-d6rgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:31:53.641974 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.641949 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:31:53.642046 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.642007 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:31:53.642406 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.642386 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f086aaa-1e01-4e56-81dc-7c4789303778" (UID: "0f086aaa-1e01-4e56-81dc-7c4789303778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:31:53.740943 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.740914 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:53.740943 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.740941 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6rgk\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-kube-api-access-d6rgk\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:53.741129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.740952 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:53.741129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.740967 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:53.741129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.740976 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:53.741129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.740984 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:53.741129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.740992 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:53.741129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:53.740999 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f086aaa-1e01-4e56-81dc-7c4789303778-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:31:54.171298 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.171264 2557 generic.go:356] "Generic (PLEG): container finished" podID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerID="09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977" exitCode=0 Jul 27 17:31:54.171478 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.171309 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" event={"ID":"0f086aaa-1e01-4e56-81dc-7c4789303778","Type":"ContainerDied","Data":"09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977"} Jul 27 17:31:54.171478 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.171317 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" Jul 27 17:31:54.171478 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.171352 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq" event={"ID":"0f086aaa-1e01-4e56-81dc-7c4789303778","Type":"ContainerDied","Data":"cb516c515025bcf7389a44c9ad0665801c55f823f67769b174dacddbf5dab56c"} Jul 27 17:31:54.171478 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.171372 2557 scope.go:117] "RemoveContainer" containerID="09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977" Jul 27 17:31:54.179829 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.179809 2557 scope.go:117] "RemoveContainer" containerID="bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d" Jul 27 17:31:54.187482 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.187464 2557 scope.go:117] "RemoveContainer" containerID="09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977" Jul 27 17:31:54.187746 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:31:54.187727 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977\": container with ID starting with 09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977 not found: ID does not exist" containerID="09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977" Jul 27 17:31:54.187793 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.187754 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977"} err="failed to get container status \"09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977\": rpc error: code = NotFound desc = could not find container \"09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977\": container with ID starting with 09afa1ececa971c900ab1f65bbb42f57896865e0d2550c6d66a2f09fd91dc977 not found: ID does not exist" Jul 27 17:31:54.187793 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.187791 2557 scope.go:117] "RemoveContainer" containerID="bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d" Jul 27 17:31:54.188039 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:31:54.188020 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d\": container with ID starting with bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d not found: ID does not exist" containerID="bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d" Jul 27 17:31:54.188092 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.188045 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d"} err="failed to get container status \"bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d\": rpc error: code = NotFound desc = could not find container \"bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d\": container with ID starting with bee49dc3a7f9b3462001689a651cdc2b006b0a979caca66c7267a284b9d6333d not found: ID does not exist" Jul 27 17:31:54.203935 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.203911 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq"] Jul 27 17:31:54.205443 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:54.205425 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-18117344-openshift-default-7667c5d984-87qbq"] Jul 27 17:31:55.169071 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:55.169039 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:55.294301 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:55.294258 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:55.419448 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:55.419355 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:55.538380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:55.538344 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:56.063978 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:31:56.063944 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" path="/var/lib/kubelet/pods/0f086aaa-1e01-4e56-81dc-7c4789303778/volumes" Jul 27 17:32:00.741912 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:00.741875 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:03.728789 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:03.728754 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:03.829790 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:03.829729 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:08.429501 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.429468 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:08.719591 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.719502 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h"] Jul 27 17:32:08.719998 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.719983 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" Jul 27 17:32:08.720048 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.720000 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" Jul 27 17:32:08.720048 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.720013 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca3a202-c921-4464-be88-fc8316f32a07" containerName="payload-pre-processing" Jul 27 17:32:08.720048 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.720022 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca3a202-c921-4464-be88-fc8316f32a07" containerName="payload-pre-processing" Jul 27 17:32:08.720048 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.720032 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" Jul 27 17:32:08.720048 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.720042 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" Jul 27 17:32:08.720243 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.720134 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca3a202-c921-4464-be88-fc8316f32a07" containerName="payload-pre-processing" Jul 27 17:32:08.720243 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.720146 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" Jul 27 17:32:08.720243 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.720159 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f086aaa-1e01-4e56-81dc-7c4789303778" containerName="istio-proxy" Jul 27 17:32:08.723475 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.723456 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.725710 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.725689 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-736d5a21-openshift-default-dockercfg-ffzcz\"" Jul 27 17:32:08.734327 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.734305 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h"] Jul 27 17:32:08.737425 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.737401 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:08.771688 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.771651 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-token\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.771853 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.771717 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-socket\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.771853 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.771751 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-podinfo\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.771853 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.771785 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-data\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.772008 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.771879 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istiod-ca-cert\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.772008 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.771937 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjns\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-kube-api-access-fcjns\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.772112 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.772028 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-certs\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.772112 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.772067 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-credential-socket\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.772112 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.772099 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-envoy\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873269 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjns\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-kube-api-access-fcjns\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-certs\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873321 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-credential-socket\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873349 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-envoy\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-token\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-socket\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873444 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-podinfo\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873469 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-data\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873539 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istiod-ca-cert\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873752 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-certs\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873948 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873804 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-credential-socket\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873948 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873860 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-socket\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.873948 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.873923 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-data\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.874406 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.874389 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istiod-ca-cert\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.875582 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.875558 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-envoy\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.876116 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.876101 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-podinfo\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.881665 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.881636 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjns\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-kube-api-access-fcjns\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:08.881743 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:08.881684 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-token\") pod \"e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:09.034565 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:09.034469 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:09.101208 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:09.101137 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:09.190510 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:32:09.190481 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e743ea4_ed43_4a22_b5b0_d951f102ecf2.slice/crio-0857bfa62cc491da5463c052d250914ef0a7b76d45e6ebf374b3fcfb81345fcd WatchSource:0}: Error finding container 0857bfa62cc491da5463c052d250914ef0a7b76d45e6ebf374b3fcfb81345fcd: Status 404 returned error can't find the container with id 0857bfa62cc491da5463c052d250914ef0a7b76d45e6ebf374b3fcfb81345fcd Jul 27 17:32:09.192531 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:09.192492 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:32:09.192619 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:09.192568 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:32:09.192619 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:09.192610 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:32:09.196158 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:09.196125 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h"] Jul 27 17:32:09.240702 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:09.240669 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" event={"ID":"1e743ea4-ed43-4a22-b5b0-d951f102ecf2","Type":"ContainerStarted","Data":"0857bfa62cc491da5463c052d250914ef0a7b76d45e6ebf374b3fcfb81345fcd"} Jul 27 17:32:10.247110 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:10.247069 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" event={"ID":"1e743ea4-ed43-4a22-b5b0-d951f102ecf2","Type":"ContainerStarted","Data":"cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2"} Jul 27 17:32:10.272474 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:10.272415 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" podStartSLOduration=2.272394918 podStartE2EDuration="2.272394918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:10.2717641 +0000 UTC m=+2126.742090296" watchObservedRunningTime="2026-07-27 17:32:10.272394918 +0000 UTC m=+2126.742721114" Jul 27 17:32:11.035502 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:11.035467 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:11.040713 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:11.040687 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:11.251112 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:11.251082 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:11.251998 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:11.251980 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:12.191272 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:12.191229 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:12.297901 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:12.297861 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:14.070977 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:14.070933 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:14.136862 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:14.136804 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:16.664232 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:16.664199 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:26.252573 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:26.252536 2557 patch_prober.go:28] interesting pod/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 27 17:32:26.253095 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:26.252592 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 27 17:32:26.328389 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:26.328354 2557 generic.go:356] "Generic (PLEG): container finished" podID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerID="cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2" exitCode=0 Jul 27 17:32:26.328389 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:26.328397 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" event={"ID":"1e743ea4-ed43-4a22-b5b0-d951f102ecf2","Type":"ContainerDied","Data":"cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2"} Jul 27 17:32:26.328811 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:26.328796 2557 scope.go:117] "RemoveContainer" containerID="cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2" Jul 27 17:32:26.329787 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:26.329756 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:32:26.329892 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:26.329830 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:32:26.329892 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:26.329863 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:32:27.334860 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:27.334819 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" event={"ID":"1e743ea4-ed43-4a22-b5b0-d951f102ecf2","Type":"ContainerStarted","Data":"f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433"} Jul 27 17:32:28.035576 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:28.035531 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:28.035766 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:28.035744 2557 patch_prober.go:28] interesting pod/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 27 17:32:28.035827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:28.035805 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 27 17:32:29.035790 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:29.035753 2557 patch_prober.go:28] interesting pod/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 27 17:32:29.036214 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:29.035811 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 27 17:32:30.034855 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:30.034818 2557 patch_prober.go:28] interesting pod/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 27 17:32:30.035033 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:30.034875 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 27 17:32:31.035191 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:31.035153 2557 patch_prober.go:28] interesting pod/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 27 17:32:31.035602 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:31.035223 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 27 17:32:32.039403 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:32.039371 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:32.039900 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:32.039742 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:32.040511 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:32.040481 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:32:34.798588 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:34.798550 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:47.561491 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:32:47.561452 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:17.072513 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:33:17.072478 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:17.197285 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:33:17.197252 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:17.296283 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:33:17.296248 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:21.343380 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:33:21.343343 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:24.446460 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:33:24.446426 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:29.479014 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:33:29.478984 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:03.001538 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:03.001507 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:05.983214 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:05.983120 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:06.861411 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:06.861383 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:12.917513 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:12.917479 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:18.998007 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:18.997970 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:19.100086 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:19.100052 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:19.132418 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:19.132383 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h"] Jul 27 17:34:19.132652 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:19.132626 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" containerID="cri-o://f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433" gracePeriod=30 Jul 27 17:34:19.168926 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:19.168891 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:19.382613 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:19.382585 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:20.507725 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:20.507691 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:20.602678 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:20.602644 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:20.734365 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:20.734329 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:21.486247 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:21.486210 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:22.201337 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:22.201261 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:22.301229 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:22.301200 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:23.057120 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:23.057074 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:24.384548 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.384524 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:34:24.462481 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462384 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-data\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.462481 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462447 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-envoy\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.462481 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462476 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-credential-socket\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.462792 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462509 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-socket\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.462792 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462543 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-podinfo\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.462792 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462569 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcjns\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-kube-api-access-fcjns\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.462792 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462613 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-certs\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.462792 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462680 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-token\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.462792 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462725 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istiod-ca-cert\") pod \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\" (UID: \"1e743ea4-ed43-4a22-b5b0-d951f102ecf2\") " Jul 27 17:34:24.463087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462903 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:34:24.463087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462931 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:34:24.463087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.462999 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:34:24.463087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.463031 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.463087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.463051 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.463333 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.463228 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:34:24.463468 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.463444 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:34:24.464879 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.464850 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:34:24.465049 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.465029 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-kube-api-access-fcjns" (OuterVolumeSpecName: "kube-api-access-fcjns") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "kube-api-access-fcjns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:34:24.465216 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.465190 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:34:24.465216 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.465206 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e743ea4-ed43-4a22-b5b0-d951f102ecf2" (UID: "1e743ea4-ed43-4a22-b5b0-d951f102ecf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:34:24.564140 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.564101 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.564140 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.564132 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.564140 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.564141 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.564477 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.564151 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcjns\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-kube-api-access-fcjns\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.564477 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.564161 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.564477 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.564169 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.564477 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.564201 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e743ea4-ed43-4a22-b5b0-d951f102ecf2-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:34:24.858659 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.858571 2557 generic.go:356] "Generic (PLEG): container finished" podID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerID="f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433" exitCode=0 Jul 27 17:34:24.858659 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.858623 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" event={"ID":"1e743ea4-ed43-4a22-b5b0-d951f102ecf2","Type":"ContainerDied","Data":"f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433"} Jul 27 17:34:24.858659 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.858638 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" Jul 27 17:34:24.858876 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.858695 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h" event={"ID":"1e743ea4-ed43-4a22-b5b0-d951f102ecf2","Type":"ContainerDied","Data":"0857bfa62cc491da5463c052d250914ef0a7b76d45e6ebf374b3fcfb81345fcd"} Jul 27 17:34:24.858876 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.858718 2557 scope.go:117] "RemoveContainer" containerID="f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433" Jul 27 17:34:24.870667 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.870628 2557 scope.go:117] "RemoveContainer" containerID="cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2" Jul 27 17:34:24.879449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.879429 2557 scope.go:117] "RemoveContainer" containerID="f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433" Jul 27 17:34:24.879696 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:34:24.879675 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433\": container with ID starting with f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433 not found: ID does not exist" containerID="f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433" Jul 27 17:34:24.879741 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.879706 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433"} err="failed to get container status \"f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433\": rpc error: code = NotFound desc = could not find container \"f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433\": container with ID starting with f21dbe4fdd4e93882d4ae65cb32f44e30344e7f3cd591e33a3deba53f8933433 not found: ID does not exist" Jul 27 17:34:24.879741 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.879724 2557 scope.go:117] "RemoveContainer" containerID="cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2" Jul 27 17:34:24.879970 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:34:24.879953 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2\": container with ID starting with cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2 not found: ID does not exist" containerID="cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2" Jul 27 17:34:24.880023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.879976 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2"} err="failed to get container status \"cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2\": rpc error: code = NotFound desc = could not find container \"cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2\": container with ID starting with cf10563ac1b9c5a93c5bd94614d25cc447e785324c157d7b2fcd2d7f565c79d2 not found: ID does not exist" Jul 27 17:34:24.885361 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.885339 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h"] Jul 27 17:34:24.891103 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:24.891083 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-736d5a21-openshift-default-5f946c69c6-vhg4h"] Jul 27 17:34:26.063850 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:26.063817 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" path="/var/lib/kubelet/pods/1e743ea4-ed43-4a22-b5b0-d951f102ecf2/volumes" Jul 27 17:34:29.179845 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:29.179817 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:29.244251 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:29.244222 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:29.881798 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:29.881768 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:37.525594 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:37.525544 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:37.637473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:37.637437 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:37.736327 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:37.736295 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:45.078516 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:45.078487 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:45.148880 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:45.148851 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:48.293069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:48.293030 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:48.651766 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:34:48.651734 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:09.593306 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:09.593267 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:16.487481 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:16.487446 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:16.563591 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:16.563553 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:16.848193 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:16.848067 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:17.206030 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.205986 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92"] Jul 27 17:35:17.206435 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.206418 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" Jul 27 17:35:17.206494 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.206437 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" Jul 27 17:35:17.206494 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.206448 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" Jul 27 17:35:17.206494 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.206454 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" Jul 27 17:35:17.206594 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.206514 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" Jul 27 17:35:17.208502 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.208485 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.211913 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.211896 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a6f598-openshift-default-dockercfg-shkzt\"" Jul 27 17:35:17.220979 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.220957 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istiod-ca-cert\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.221068 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.220996 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-socket\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.221068 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.221053 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-credential-socket\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.221148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.221070 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-certs\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.221148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.221092 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-token\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.221148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.221114 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-podinfo\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.221301 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.221148 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8w7\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-kube-api-access-sb8w7\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.221301 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.221200 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-envoy\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.221301 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.221224 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-data\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.228925 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.228902 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92"] Jul 27 17:35:17.243353 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.243331 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:17.322211 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322150 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-credential-socket\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322211 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322207 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-certs\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322211 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322226 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-token\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322251 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-podinfo\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8w7\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-kube-api-access-sb8w7\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-envoy\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322351 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-data\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322485 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istiod-ca-cert\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322802 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322534 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-socket\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322802 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322584 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-credential-socket\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322802 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322660 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-certs\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.322951 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322905 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-data\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.323009 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.322973 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-socket\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.323251 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.323157 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istiod-ca-cert\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.324679 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.324659 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-envoy\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.324788 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.324772 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-podinfo\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.334563 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.334544 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-token\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.335127 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.335107 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8w7\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-kube-api-access-sb8w7\") pod \"e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.521906 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.521819 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:17.584042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.583991 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:17.665417 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.665388 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92"] Jul 27 17:35:17.667415 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:35:17.667390 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c0f05b_7b15_4240_b5e4_6b9255a7540e.slice/crio-b205bb19bc99f25138d8fcf9e7b68ba0c78fc28394adbcb2c1d6e59692f3a257 WatchSource:0}: Error finding container b205bb19bc99f25138d8fcf9e7b68ba0c78fc28394adbcb2c1d6e59692f3a257: Status 404 returned error can't find the container with id b205bb19bc99f25138d8fcf9e7b68ba0c78fc28394adbcb2c1d6e59692f3a257 Jul 27 17:35:17.669527 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.669493 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:17.669629 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.669576 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:17.669629 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:17.669623 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:18.103441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:18.103399 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" event={"ID":"95c0f05b-7b15-4240-b5e4-6b9255a7540e","Type":"ContainerStarted","Data":"bf6c901e2263687806a6c8e288df157b26445f837e8049eff71c0da3a5f3bbb7"} Jul 27 17:35:18.103630 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:18.103473 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" event={"ID":"95c0f05b-7b15-4240-b5e4-6b9255a7540e","Type":"ContainerStarted","Data":"b205bb19bc99f25138d8fcf9e7b68ba0c78fc28394adbcb2c1d6e59692f3a257"} Jul 27 17:35:18.127134 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:18.127078 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" podStartSLOduration=1.12705665 podStartE2EDuration="1.12705665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:18.125076715 +0000 UTC m=+2314.595402910" watchObservedRunningTime="2026-07-27 17:35:18.12705665 +0000 UTC m=+2314.597382849" Jul 27 17:35:18.522303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:18.522261 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:18.527026 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:18.526998 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:19.108497 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:19.108465 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:19.109654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:19.109633 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:20.093619 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:20.093578 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:20.194687 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:20.194646 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:20.258653 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:20.258615 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:20.360106 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:20.359996 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:20.890276 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:20.890239 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:20.988074 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:20.988038 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:24.564577 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:24.564540 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:24.887900 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:24.887828 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:25.153710 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.153620 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr"] Jul 27 17:35:25.154084 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.154069 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e743ea4-ed43-4a22-b5b0-d951f102ecf2" containerName="istio-proxy" Jul 27 17:35:25.156171 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.156148 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.159213 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.159169 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7a55cb-openshift-default-dockercfg-m6595\"" Jul 27 17:35:25.170390 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.170365 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr"] Jul 27 17:35:25.182151 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.182123 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:25.192314 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192291 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-token\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.192433 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192331 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-podinfo\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.192433 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192364 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-socket\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.192574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192449 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-credential-socket\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.192574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192486 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-certs\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.192574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43041b3-5298-4ad9-bf5d-d3678702cd11-istiod-ca-cert\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.192574 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192550 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5dx\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-kube-api-access-vt5dx\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.192792 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192590 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-data\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.192792 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.192706 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-envoy\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.293885 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.293853 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-token\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.293910 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-podinfo\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.293950 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-socket\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.293977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-credential-socket\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294027 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-certs\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294328 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294086 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43041b3-5298-4ad9-bf5d-d3678702cd11-istiod-ca-cert\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294328 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5dx\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-kube-api-access-vt5dx\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294328 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294158 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-data\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294328 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294229 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-envoy\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294550 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294356 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-socket\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294550 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294417 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-certs\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294550 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294529 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-data\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.294798 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.294774 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-credential-socket\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.295025 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.295005 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43041b3-5298-4ad9-bf5d-d3678702cd11-istiod-ca-cert\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.296610 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.296587 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-podinfo\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.296699 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.296659 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-envoy\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.302710 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.302690 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-token\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.302811 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.302703 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5dx\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-kube-api-access-vt5dx\") pod \"e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.467184 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.467079 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:25.553691 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.553658 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:25.610497 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.610463 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr"] Jul 27 17:35:25.614243 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:35:25.614197 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43041b3_5298_4ad9_bf5d_d3678702cd11.slice/crio-5f74cbb9a418514ca7673faa8942277e2dab06dae41209a2beef68062217ab9e WatchSource:0}: Error finding container 5f74cbb9a418514ca7673faa8942277e2dab06dae41209a2beef68062217ab9e: Status 404 returned error can't find the container with id 5f74cbb9a418514ca7673faa8942277e2dab06dae41209a2beef68062217ab9e Jul 27 17:35:25.617263 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.617222 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:25.617397 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.617377 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:25.617456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:25.617424 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:26.144069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:26.144034 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" event={"ID":"b43041b3-5298-4ad9-bf5d-d3678702cd11","Type":"ContainerStarted","Data":"00f140bf4ab2f5b13d8222f1fe6a41bec02806e35182b930397e6d2ac8a0e379"} Jul 27 17:35:26.144311 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:26.144087 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" event={"ID":"b43041b3-5298-4ad9-bf5d-d3678702cd11","Type":"ContainerStarted","Data":"5f74cbb9a418514ca7673faa8942277e2dab06dae41209a2beef68062217ab9e"} Jul 27 17:35:26.165340 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:26.165271 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" podStartSLOduration=1.165246842 podStartE2EDuration="1.165246842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:26.164212914 +0000 UTC m=+2322.634539111" watchObservedRunningTime="2026-07-27 17:35:26.165246842 +0000 UTC m=+2322.635573040" Jul 27 17:35:26.469367 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:26.469267 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:26.474222 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:26.474187 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:27.152452 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:27.152415 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:27.156836 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:27.156806 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:28.072615 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:28.072577 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:28.160614 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:28.160569 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:28.222303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:28.222267 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:28.520148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:28.520112 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:28.799869 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:28.799755 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:28.894337 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:28.894294 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:29.018861 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.018824 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z"] Jul 27 17:35:29.021159 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.021141 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:29.023536 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.023518 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7a55cb\"" Jul 27 17:35:29.023638 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.023539 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7a55cb-dockercfg-tlwzk\"" Jul 27 17:35:29.033426 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.033403 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z"] Jul 27 17:35:29.131868 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.131831 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpnrp\" (UniqueName: \"kubernetes.io/projected/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-kube-api-access-kpnrp\") pod \"payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z\" (UID: \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:29.132031 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.131947 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z\" (UID: \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:29.232984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.232940 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z\" (UID: \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:29.233456 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.233304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnrp\" (UniqueName: \"kubernetes.io/projected/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-kube-api-access-kpnrp\") pod \"payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z\" (UID: \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:29.233732 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.233715 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z\" (UID: \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:29.241656 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.241625 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnrp\" (UniqueName: \"kubernetes.io/projected/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-kube-api-access-kpnrp\") pod \"payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z\" (UID: \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:29.332065 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.332025 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:29.521037 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:29.520976 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z"] Jul 27 17:35:29.523305 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:35:29.523264 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd4a5a0_e73c_4187_90bc_0cf52eaf077e.slice/crio-4a39d6ca3161cafa6390fc065bac2f3c673ab4e06b409a0e49149db81d5522e8 WatchSource:0}: Error finding container 4a39d6ca3161cafa6390fc065bac2f3c673ab4e06b409a0e49149db81d5522e8: Status 404 returned error can't find the container with id 4a39d6ca3161cafa6390fc065bac2f3c673ab4e06b409a0e49149db81d5522e8 Jul 27 17:35:30.176000 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:30.175965 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" event={"ID":"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e","Type":"ContainerStarted","Data":"4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac"} Jul 27 17:35:30.176194 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:30.176016 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" event={"ID":"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e","Type":"ContainerStarted","Data":"4a39d6ca3161cafa6390fc065bac2f3c673ab4e06b409a0e49149db81d5522e8"} Jul 27 17:35:30.176194 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:30.176114 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:30.209042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:30.208986 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" podStartSLOduration=1.208969085 podStartE2EDuration="1.208969085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:30.205891462 +0000 UTC m=+2326.676217657" watchObservedRunningTime="2026-07-27 17:35:30.208969085 +0000 UTC m=+2326.679295279" Jul 27 17:35:32.515282 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:32.515253 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:32.677525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:32.677491 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:33.192840 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:33.192808 2557 generic.go:356] "Generic (PLEG): container finished" podID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerID="bf6c901e2263687806a6c8e288df157b26445f837e8049eff71c0da3a5f3bbb7" exitCode=0 Jul 27 17:35:33.193015 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:33.192880 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" event={"ID":"95c0f05b-7b15-4240-b5e4-6b9255a7540e","Type":"ContainerDied","Data":"bf6c901e2263687806a6c8e288df157b26445f837e8049eff71c0da3a5f3bbb7"} Jul 27 17:35:33.193401 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:33.193383 2557 scope.go:117] "RemoveContainer" containerID="bf6c901e2263687806a6c8e288df157b26445f837e8049eff71c0da3a5f3bbb7" Jul 27 17:35:33.194542 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:33.194515 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:33.194601 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:33.194582 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:33.194637 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:33.194621 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:34.108466 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:34.108422 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:34.199908 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:34.199870 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" event={"ID":"95c0f05b-7b15-4240-b5e4-6b9255a7540e","Type":"ContainerStarted","Data":"5eea61c3d70d8f25c089c38ee6a4fbea66c96a76a7d71564fffacc858119f521"} Jul 27 17:35:34.522773 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:34.522740 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:34.523994 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:34.523967 2557 patch_prober.go:28] interesting pod/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 27 17:35:34.524127 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:34.524016 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 27 17:35:35.523644 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:35.523216 2557 patch_prober.go:28] interesting pod/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 27 17:35:35.523644 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:35.523285 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 27 17:35:36.523004 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:36.522966 2557 patch_prober.go:28] interesting pod/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 27 17:35:36.523217 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:36.523024 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 27 17:35:37.523261 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:37.523129 2557 patch_prober.go:28] interesting pod/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 27 17:35:37.523261 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:37.523213 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 27 17:35:37.866138 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:37.866038 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:38.522604 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:38.522561 2557 patch_prober.go:28] interesting pod/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 27 17:35:38.522800 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:38.522625 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 27 17:35:39.527373 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:39.527339 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:39.527811 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:39.527643 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:39.528303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:39.528284 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:35:41.183989 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:41.183955 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:35:41.236554 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:41.236517 2557 generic.go:356] "Generic (PLEG): container finished" podID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerID="00f140bf4ab2f5b13d8222f1fe6a41bec02806e35182b930397e6d2ac8a0e379" exitCode=0 Jul 27 17:35:41.236741 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:41.236588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" event={"ID":"b43041b3-5298-4ad9-bf5d-d3678702cd11","Type":"ContainerDied","Data":"00f140bf4ab2f5b13d8222f1fe6a41bec02806e35182b930397e6d2ac8a0e379"} Jul 27 17:35:41.237143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:41.237124 2557 scope.go:117] "RemoveContainer" containerID="00f140bf4ab2f5b13d8222f1fe6a41bec02806e35182b930397e6d2ac8a0e379" Jul 27 17:35:41.238162 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:41.238131 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:41.238272 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:41.238232 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:41.238323 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:41.238271 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:35:42.152368 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:42.152330 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:42.244262 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:42.244221 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" event={"ID":"b43041b3-5298-4ad9-bf5d-d3678702cd11","Type":"ContainerStarted","Data":"d5b70c95c599e247aba624c0709cadc44fd4880e48524add068ffbf0b54e3b25"} Jul 27 17:35:42.468122 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:42.468021 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:42.469579 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:42.469549 2557 patch_prober.go:28] interesting pod/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 27 17:35:42.469745 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:42.469613 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 27 17:35:43.468010 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:43.467973 2557 patch_prober.go:28] interesting pod/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 27 17:35:43.468629 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:43.468594 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 27 17:35:44.469642 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:44.469602 2557 patch_prober.go:28] interesting pod/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 27 17:35:44.470111 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:44.469667 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 27 17:35:45.468450 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:45.468396 2557 patch_prober.go:28] interesting pod/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 27 17:35:45.468768 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:45.468728 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 27 17:35:46.472220 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:46.472168 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:46.472664 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:46.472464 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:46.473123 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:46.473099 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:35:56.782049 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:35:56.782014 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:36.578423 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:36.578384 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:37.514801 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:37.514772 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:48.355377 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:48.355341 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:48.452734 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:48.452705 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:48.519391 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:48.519360 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:52.713473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:52.713436 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:53.095160 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:53.095079 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:59.180710 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:59.180675 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:59.282707 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:59.282678 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:59.369646 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:59.369603 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92"] Jul 27 17:36:59.369839 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:59.369818 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" containerID="cri-o://5eea61c3d70d8f25c089c38ee6a4fbea66c96a76a7d71564fffacc858119f521" gracePeriod=30 Jul 27 17:36:59.400869 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:59.400838 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:59.651495 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:36:59.651464 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:04.631570 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.631490 2557 generic.go:356] "Generic (PLEG): container finished" podID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerID="5eea61c3d70d8f25c089c38ee6a4fbea66c96a76a7d71564fffacc858119f521" exitCode=0 Jul 27 17:37:04.631894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.631565 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" event={"ID":"95c0f05b-7b15-4240-b5e4-6b9255a7540e","Type":"ContainerDied","Data":"5eea61c3d70d8f25c089c38ee6a4fbea66c96a76a7d71564fffacc858119f521"} Jul 27 17:37:04.631894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.631626 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" event={"ID":"95c0f05b-7b15-4240-b5e4-6b9255a7540e","Type":"ContainerDied","Data":"b205bb19bc99f25138d8fcf9e7b68ba0c78fc28394adbcb2c1d6e59692f3a257"} Jul 27 17:37:04.631894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.631645 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b205bb19bc99f25138d8fcf9e7b68ba0c78fc28394adbcb2c1d6e59692f3a257" Jul 27 17:37:04.631894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.631660 2557 scope.go:117] "RemoveContainer" containerID="bf6c901e2263687806a6c8e288df157b26445f837e8049eff71c0da3a5f3bbb7" Jul 27 17:37:04.632106 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.631977 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:37:04.706876 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.706841 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-credential-socket\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.706876 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.706878 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-token\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.707129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.706904 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-podinfo\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.707129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.706926 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-data\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.707129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.706964 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb8w7\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-kube-api-access-sb8w7\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.707129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.706990 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istiod-ca-cert\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.707129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.707011 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-certs\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.707129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.707040 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-socket\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.707129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.707094 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-envoy\") pod \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\" (UID: \"95c0f05b-7b15-4240-b5e4-6b9255a7540e\") " Jul 27 17:37:04.707533 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.707160 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:04.707533 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.707404 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:04.707533 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.707483 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:04.707698 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.707562 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:37:04.708020 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.707997 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:04.708118 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.708010 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-data" (OuterVolumeSpecName: "istio-data") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:04.708118 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.708024 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:04.708118 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.708069 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:04.708118 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.708086 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:04.709307 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.709277 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-kube-api-access-sb8w7" (OuterVolumeSpecName: "kube-api-access-sb8w7") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "kube-api-access-sb8w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:37:04.709413 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.709326 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-token" (OuterVolumeSpecName: "istio-token") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:37:04.709413 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.709342 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:37:04.709485 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.709451 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95c0f05b-7b15-4240-b5e4-6b9255a7540e" (UID: "95c0f05b-7b15-4240-b5e4-6b9255a7540e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:04.809131 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.809100 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:04.809131 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.809127 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:04.809131 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.809136 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:04.809377 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.809144 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0f05b-7b15-4240-b5e4-6b9255a7540e-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:04.809377 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:04.809154 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb8w7\" (UniqueName: \"kubernetes.io/projected/95c0f05b-7b15-4240-b5e4-6b9255a7540e-kube-api-access-sb8w7\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:05.638313 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:05.638238 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92" Jul 27 17:37:05.679042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:05.679017 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92"] Jul 27 17:37:05.681794 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:05.681770 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a6f598-openshift-default-659b498b48-6mq92"] Jul 27 17:37:06.067430 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:06.065269 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" path="/var/lib/kubelet/pods/95c0f05b-7b15-4240-b5e4-6b9255a7540e/volumes" Jul 27 17:37:10.328913 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.328880 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:10.375020 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.374985 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z"] Jul 27 17:37:10.375235 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.375212 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" podUID="9dd4a5a0-e73c-4187-90bc-0cf52eaf077e" containerName="payload-pre-processing" containerID="cri-o://4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac" gracePeriod=30 Jul 27 17:37:10.403588 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.403555 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:10.551315 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.541292 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:10.637670 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.637647 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:37:10.665160 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.665132 2557 generic.go:356] "Generic (PLEG): container finished" podID="9dd4a5a0-e73c-4187-90bc-0cf52eaf077e" containerID="4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac" exitCode=0 Jul 27 17:37:10.665345 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.665196 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" Jul 27 17:37:10.665345 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.665203 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" event={"ID":"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e","Type":"ContainerDied","Data":"4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac"} Jul 27 17:37:10.665345 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.665253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z" event={"ID":"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e","Type":"ContainerDied","Data":"4a39d6ca3161cafa6390fc065bac2f3c673ab4e06b409a0e49149db81d5522e8"} Jul 27 17:37:10.665345 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.665276 2557 scope.go:117] "RemoveContainer" containerID="4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac" Jul 27 17:37:10.673528 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.673512 2557 scope.go:117] "RemoveContainer" containerID="4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac" Jul 27 17:37:10.673804 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:37:10.673780 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac\": container with ID starting with 4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac not found: ID does not exist" containerID="4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac" Jul 27 17:37:10.673895 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.673811 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac"} err="failed to get container status \"4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac\": rpc error: code = NotFound desc = could not find container \"4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac\": container with ID starting with 4292c975e82d3a3c032b2fb3e9bedd23cdcc94d4883cb1215dcd081ea969b7ac not found: ID does not exist" Jul 27 17:37:10.761534 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.761502 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-plugins-config-volume\") pod \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\" (UID: \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\") " Jul 27 17:37:10.761674 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.761561 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpnrp\" (UniqueName: \"kubernetes.io/projected/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-kube-api-access-kpnrp\") pod \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\" (UID: \"9dd4a5a0-e73c-4187-90bc-0cf52eaf077e\") " Jul 27 17:37:10.761914 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.761889 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9dd4a5a0-e73c-4187-90bc-0cf52eaf077e" (UID: "9dd4a5a0-e73c-4187-90bc-0cf52eaf077e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:37:10.763564 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.763543 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-kube-api-access-kpnrp" (OuterVolumeSpecName: "kube-api-access-kpnrp") pod "9dd4a5a0-e73c-4187-90bc-0cf52eaf077e" (UID: "9dd4a5a0-e73c-4187-90bc-0cf52eaf077e"). InnerVolumeSpecName "kube-api-access-kpnrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:37:10.862870 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.862839 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpnrp\" (UniqueName: \"kubernetes.io/projected/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-kube-api-access-kpnrp\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:10.862870 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.862867 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:10.989588 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.989556 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z"] Jul 27 17:37:10.994064 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:10.994042 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7a55cb-845899f6b4-lx95z"] Jul 27 17:37:12.063931 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:12.063892 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dd4a5a0-e73c-4187-90bc-0cf52eaf077e" path="/var/lib/kubelet/pods/9dd4a5a0-e73c-4187-90bc-0cf52eaf077e/volumes" Jul 27 17:37:14.856943 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:14.856904 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:15.226455 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:15.226422 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:21.280115 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:21.280076 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:21.520209 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:21.520158 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr"] Jul 27 17:37:21.520432 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:21.520407 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" containerID="cri-o://d5b70c95c599e247aba624c0709cadc44fd4880e48524add068ffbf0b54e3b25" gracePeriod=30 Jul 27 17:37:22.393688 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.393658 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q"] Jul 27 17:37:22.394051 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394007 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" Jul 27 17:37:22.394051 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394017 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" Jul 27 17:37:22.394051 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394026 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd4a5a0-e73c-4187-90bc-0cf52eaf077e" containerName="payload-pre-processing" Jul 27 17:37:22.394051 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394032 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd4a5a0-e73c-4187-90bc-0cf52eaf077e" containerName="payload-pre-processing" Jul 27 17:37:22.394051 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394046 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" Jul 27 17:37:22.394051 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394052 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" Jul 27 17:37:22.394280 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394096 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" Jul 27 17:37:22.394280 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394107 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c0f05b-7b15-4240-b5e4-6b9255a7540e" containerName="istio-proxy" Jul 27 17:37:22.394280 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.394114 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd4a5a0-e73c-4187-90bc-0cf52eaf077e" containerName="payload-pre-processing" Jul 27 17:37:22.397135 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.397111 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.399832 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.399811 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f6e958-openshift-default-dockercfg-9kg9c\"" Jul 27 17:37:22.414607 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.414580 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q"] Jul 27 17:37:22.560122 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560085 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-podinfo\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.560315 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560149 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-certs\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.560315 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560206 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-token\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.560315 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560236 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208e7070-a527-4c0a-8ffc-7d459dbb5006-istiod-ca-cert\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.560315 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560268 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-data\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.560315 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560308 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-credential-socket\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.560560 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560340 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-socket\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.560560 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560395 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5wck\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-kube-api-access-n5wck\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.560560 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.560433 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-envoy\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.661500 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661409 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-podinfo\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.661500 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661465 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-certs\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.661500 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661484 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-token\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.661789 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661506 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208e7070-a527-4c0a-8ffc-7d459dbb5006-istiod-ca-cert\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.661789 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661527 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-data\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.661789 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661554 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-credential-socket\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.661789 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661577 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-socket\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.661789 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661615 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5wck\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-kube-api-access-n5wck\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.662040 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661907 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-certs\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.662040 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661942 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-credential-socket\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.662040 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.661995 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-envoy\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.662040 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.662013 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-data\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.662259 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.662047 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-socket\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.662323 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.662286 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208e7070-a527-4c0a-8ffc-7d459dbb5006-istiod-ca-cert\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.664080 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.664059 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-envoy\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.664348 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.664328 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-podinfo\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.670230 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.670203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-token\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.670953 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.670926 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5wck\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-kube-api-access-n5wck\") pod \"e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.708276 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.708241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:22.751981 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.751946 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:22.849933 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.849910 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q"] Jul 27 17:37:22.852435 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:37:22.852409 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208e7070_a527_4c0a_8ffc_7d459dbb5006.slice/crio-728bb215384451de2d3ec442a9e3a6a76a4f8c609224c0d705da82f80e1f198a WatchSource:0}: Error finding container 728bb215384451de2d3ec442a9e3a6a76a4f8c609224c0d705da82f80e1f198a: Status 404 returned error can't find the container with id 728bb215384451de2d3ec442a9e3a6a76a4f8c609224c0d705da82f80e1f198a Jul 27 17:37:22.854702 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.854664 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:22.854786 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.854748 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:22.854825 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:22.854792 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:23.736356 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:23.736316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" event={"ID":"208e7070-a527-4c0a-8ffc-7d459dbb5006","Type":"ContainerStarted","Data":"c5dc7f0d475699dcef469dd9ffade7ebac9fd4efb0445578d04eeab386b60eb3"} Jul 27 17:37:23.736704 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:23.736403 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" event={"ID":"208e7070-a527-4c0a-8ffc-7d459dbb5006","Type":"ContainerStarted","Data":"728bb215384451de2d3ec442a9e3a6a76a4f8c609224c0d705da82f80e1f198a"} Jul 27 17:37:23.759884 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:23.759833 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" podStartSLOduration=1.7598183010000001 podStartE2EDuration="1.759818301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:23.758740483 +0000 UTC m=+2440.229066680" watchObservedRunningTime="2026-07-27 17:37:23.759818301 +0000 UTC m=+2440.230144493" Jul 27 17:37:24.708895 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:24.708860 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:24.713509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:24.713485 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:24.741472 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:24.741440 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:24.742234 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:24.742216 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:25.748004 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:25.747967 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:25.842859 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:25.842778 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:25.911346 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:25.911313 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:26.011422 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.011332 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:26.460550 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.460512 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:26.555664 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.555639 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:26.758321 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.758222 2557 generic.go:356] "Generic (PLEG): container finished" podID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerID="d5b70c95c599e247aba624c0709cadc44fd4880e48524add068ffbf0b54e3b25" exitCode=0 Jul 27 17:37:26.758321 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.758300 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" event={"ID":"b43041b3-5298-4ad9-bf5d-d3678702cd11","Type":"ContainerDied","Data":"d5b70c95c599e247aba624c0709cadc44fd4880e48524add068ffbf0b54e3b25"} Jul 27 17:37:26.758854 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.758371 2557 scope.go:117] "RemoveContainer" containerID="00f140bf4ab2f5b13d8222f1fe6a41bec02806e35182b930397e6d2ac8a0e379" Jul 27 17:37:26.791022 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.790998 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:37:26.799643 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.799612 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-token\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.799769 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.799651 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-certs\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.799769 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.799675 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-socket\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.799769 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.799698 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt5dx\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-kube-api-access-vt5dx\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.799769 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.799738 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-podinfo\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.799984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.799773 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-data\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.799984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.799808 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-envoy\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.800101 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.799979 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:26.800101 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800012 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:26.800101 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800067 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-credential-socket\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.800101 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800100 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43041b3-5298-4ad9-bf5d-d3678702cd11-istiod-ca-cert\") pod \"b43041b3-5298-4ad9-bf5d-d3678702cd11\" (UID: \"b43041b3-5298-4ad9-bf5d-d3678702cd11\") " Jul 27 17:37:26.800816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800372 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:26.800816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800590 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43041b3-5298-4ad9-bf5d-d3678702cd11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:37:26.800816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800603 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:26.800816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800625 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:26.800816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800639 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:26.801094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.800892 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-data" (OuterVolumeSpecName: "istio-data") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:26.802112 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.802065 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-token" (OuterVolumeSpecName: "istio-token") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:37:26.802374 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.802351 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:37:26.802465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.802447 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-kube-api-access-vt5dx" (OuterVolumeSpecName: "kube-api-access-vt5dx") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "kube-api-access-vt5dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:37:26.803066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.803046 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b43041b3-5298-4ad9-bf5d-d3678702cd11" (UID: "b43041b3-5298-4ad9-bf5d-d3678702cd11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:37:26.901334 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.901293 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt5dx\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-kube-api-access-vt5dx\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:26.901334 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.901327 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:26.901334 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.901340 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:26.901568 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.901350 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:26.901568 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.901361 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b43041b3-5298-4ad9-bf5d-d3678702cd11-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:26.901568 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:26.901371 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b43041b3-5298-4ad9-bf5d-d3678702cd11-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:37:27.764632 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:27.764605 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" Jul 27 17:37:27.765096 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:27.764600 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr" event={"ID":"b43041b3-5298-4ad9-bf5d-d3678702cd11","Type":"ContainerDied","Data":"5f74cbb9a418514ca7673faa8942277e2dab06dae41209a2beef68062217ab9e"} Jul 27 17:37:27.765096 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:27.764760 2557 scope.go:117] "RemoveContainer" containerID="d5b70c95c599e247aba624c0709cadc44fd4880e48524add068ffbf0b54e3b25" Jul 27 17:37:27.786315 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:27.786290 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr"] Jul 27 17:37:27.790536 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:27.790514 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7a55cb-openshift-default-54dfdb87f4-qcjtr"] Jul 27 17:37:28.070726 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:28.070644 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" path="/var/lib/kubelet/pods/b43041b3-5298-4ad9-bf5d-d3678702cd11/volumes" Jul 27 17:37:30.147778 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.147740 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:30.645392 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.645359 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:30.945269 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.945166 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865"] Jul 27 17:37:30.945633 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.945615 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" Jul 27 17:37:30.945722 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.945636 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" Jul 27 17:37:30.945722 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.945717 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" Jul 27 17:37:30.950407 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.950385 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:30.952401 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.952379 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-05e104-openshift-default-dockercfg-kzc87\"" Jul 27 17:37:30.963029 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.963006 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:30.963782 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:30.963761 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865"] Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041580 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istiod-ca-cert\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041665 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-socket\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041692 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-credential-socket\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041721 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfgrd\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-kube-api-access-rfgrd\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041747 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-data\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041778 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-certs\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041817 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-podinfo\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041844 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-token\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.042023 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.041912 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-envoy\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.142899 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.142863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istiod-ca-cert\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.142934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-socket\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.142962 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-credential-socket\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143069 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.142988 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfgrd\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-kube-api-access-rfgrd\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143284 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143117 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-data\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143284 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-certs\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143400 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-podinfo\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143400 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-token\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143531 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143477 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-socket\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143531 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143475 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-envoy\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143531 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-credential-socket\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143725 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143698 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istiod-ca-cert\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143831 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143803 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-certs\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.143831 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.143815 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-data\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.145778 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.145745 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-envoy\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.146268 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.146160 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-podinfo\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.150579 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.150551 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfgrd\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-kube-api-access-rfgrd\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.151028 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.151006 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-token\") pod \"e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.260492 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.260411 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:31.285788 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.285757 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:31.409523 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.409501 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865"] Jul 27 17:37:31.411604 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:37:31.411576 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7437bafa_405e_4c78_a0f9_bbf2fdf48aec.slice/crio-4e20eeb58dcbedefbab0f60905f3cc39227f0f8c781f74bc08acdf3dc0a776ad WatchSource:0}: Error finding container 4e20eeb58dcbedefbab0f60905f3cc39227f0f8c781f74bc08acdf3dc0a776ad: Status 404 returned error can't find the container with id 4e20eeb58dcbedefbab0f60905f3cc39227f0f8c781f74bc08acdf3dc0a776ad Jul 27 17:37:31.413857 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.413820 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:31.413926 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.413887 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:31.413926 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.413915 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:31.786521 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.786482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" event={"ID":"7437bafa-405e-4c78-a0f9-bbf2fdf48aec","Type":"ContainerStarted","Data":"2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785"} Jul 27 17:37:31.786714 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.786548 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" event={"ID":"7437bafa-405e-4c78-a0f9-bbf2fdf48aec","Type":"ContainerStarted","Data":"4e20eeb58dcbedefbab0f60905f3cc39227f0f8c781f74bc08acdf3dc0a776ad"} Jul 27 17:37:31.811706 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:31.811659 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" podStartSLOduration=1.811644073 podStartE2EDuration="1.811644073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:31.809954548 +0000 UTC m=+2448.280280744" watchObservedRunningTime="2026-07-27 17:37:31.811644073 +0000 UTC m=+2448.281970264" Jul 27 17:37:32.266610 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:32.261528 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:32.275768 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:32.275730 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:32.791432 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:32.791394 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:32.792410 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:32.792388 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:33.704726 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:33.704689 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:33.797275 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:33.797236 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:33.938042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:33.938007 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:34.166135 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:34.166092 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:34.513417 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:34.513333 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:34.613973 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:34.613941 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:38.164639 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:38.164598 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:38.823050 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:38.823020 2557 generic.go:356] "Generic (PLEG): container finished" podID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerID="c5dc7f0d475699dcef469dd9ffade7ebac9fd4efb0445578d04eeab386b60eb3" exitCode=0 Jul 27 17:37:38.823233 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:38.823071 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" event={"ID":"208e7070-a527-4c0a-8ffc-7d459dbb5006","Type":"ContainerDied","Data":"c5dc7f0d475699dcef469dd9ffade7ebac9fd4efb0445578d04eeab386b60eb3"} Jul 27 17:37:38.823458 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:38.823443 2557 scope.go:117] "RemoveContainer" containerID="c5dc7f0d475699dcef469dd9ffade7ebac9fd4efb0445578d04eeab386b60eb3" Jul 27 17:37:38.824424 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:38.824394 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:38.824516 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:38.824463 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:38.824516 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:38.824496 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:39.741617 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:39.741581 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:39.829083 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:39.829043 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" event={"ID":"208e7070-a527-4c0a-8ffc-7d459dbb5006","Type":"ContainerStarted","Data":"cc7a33568e56fb3e6ae00e27abefc1fa5c048147f04423c1e6a35dd91e50ccfe"} Jul 27 17:37:40.709237 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:40.709203 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:40.710350 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:40.710324 2557 patch_prober.go:28] interesting pod/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 27 17:37:40.710450 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:40.710378 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 27 17:37:41.709458 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:41.709425 2557 patch_prober.go:28] interesting pod/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 27 17:37:41.709933 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:41.709484 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 27 17:37:42.709073 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:42.709039 2557 patch_prober.go:28] interesting pod/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 27 17:37:42.709309 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:42.709097 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 27 17:37:43.720939 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:43.720902 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:43.721421 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:43.721308 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:43.721988 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:43.721967 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:37:50.890683 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:50.890652 2557 generic.go:356] "Generic (PLEG): container finished" podID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerID="2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785" exitCode=0 Jul 27 17:37:50.891213 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:50.890703 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" event={"ID":"7437bafa-405e-4c78-a0f9-bbf2fdf48aec","Type":"ContainerDied","Data":"2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785"} Jul 27 17:37:50.891213 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:50.891170 2557 scope.go:117] "RemoveContainer" containerID="2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785" Jul 27 17:37:50.892253 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:50.892223 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:50.892318 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:50.892295 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:50.892354 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:50.892325 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:37:51.902537 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:51.902498 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" event={"ID":"7437bafa-405e-4c78-a0f9-bbf2fdf48aec","Type":"ContainerStarted","Data":"860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8"} Jul 27 17:37:52.261257 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:52.260885 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:52.261257 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:52.261024 2557 patch_prober.go:28] interesting pod/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 27 17:37:52.261257 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:52.261069 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 27 17:37:53.261222 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:53.261157 2557 patch_prober.go:28] interesting pod/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 27 17:37:53.261832 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:53.261796 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 27 17:37:54.261287 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:54.261244 2557 patch_prober.go:28] interesting pod/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 27 17:37:54.261894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:54.261862 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 27 17:37:55.261116 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:55.261076 2557 patch_prober.go:28] interesting pod/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 27 17:37:55.261441 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:55.261404 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 27 17:37:56.265650 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:56.265617 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:56.266024 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:56.265947 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:37:56.266757 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:37:56.266735 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:38:01.183905 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:01.183873 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:02.385619 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:02.385589 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:02.453469 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:02.453429 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:05.525311 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:05.525278 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:06.139362 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:06.139328 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:06.508887 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:06.508804 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:12.565997 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:12.565964 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:12.681050 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:12.681018 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:12.796001 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:12.795970 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q"] Jul 27 17:38:12.796230 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:12.796205 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" containerID="cri-o://cc7a33568e56fb3e6ae00e27abefc1fa5c048147f04423c1e6a35dd91e50ccfe" gracePeriod=30 Jul 27 17:38:12.817851 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:12.817767 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:13.088299 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:13.088171 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:13.723471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:13.723435 2557 patch_prober.go:28] interesting pod/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 17:38:13.723848 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:13.723496 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:38:18.027403 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.027370 2557 generic.go:356] "Generic (PLEG): container finished" podID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerID="cc7a33568e56fb3e6ae00e27abefc1fa5c048147f04423c1e6a35dd91e50ccfe" exitCode=0 Jul 27 17:38:18.027759 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.027454 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" event={"ID":"208e7070-a527-4c0a-8ffc-7d459dbb5006","Type":"ContainerDied","Data":"cc7a33568e56fb3e6ae00e27abefc1fa5c048147f04423c1e6a35dd91e50ccfe"} Jul 27 17:38:18.027759 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.027566 2557 scope.go:117] "RemoveContainer" containerID="c5dc7f0d475699dcef469dd9ffade7ebac9fd4efb0445578d04eeab386b60eb3" Jul 27 17:38:18.063585 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.063528 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:38:18.171813 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.171783 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5wck\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-kube-api-access-n5wck\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172005 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.171822 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208e7070-a527-4c0a-8ffc-7d459dbb5006-istiod-ca-cert\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172005 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.171890 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-data\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172005 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.171951 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-certs\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172005 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.171987 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-envoy\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172250 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.172020 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-credential-socket\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172250 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.172059 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-token\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172250 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.172086 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-podinfo\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172250 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.172122 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-socket\") pod \"208e7070-a527-4c0a-8ffc-7d459dbb5006\" (UID: \"208e7070-a527-4c0a-8ffc-7d459dbb5006\") " Jul 27 17:38:18.172443 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.172382 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:18.172443 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.172398 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208e7070-a527-4c0a-8ffc-7d459dbb5006-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:38:18.172632 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.172606 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:18.172876 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.172844 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:18.173609 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.173583 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-data" (OuterVolumeSpecName: "istio-data") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:18.174330 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.174300 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-token" (OuterVolumeSpecName: "istio-token") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:38:18.174473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.174450 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-kube-api-access-n5wck" (OuterVolumeSpecName: "kube-api-access-n5wck") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "kube-api-access-n5wck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:38:18.174617 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.174591 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:38:18.174921 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.174893 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "208e7070-a527-4c0a-8ffc-7d459dbb5006" (UID: "208e7070-a527-4c0a-8ffc-7d459dbb5006"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:18.273709 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273681 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:18.273709 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273704 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:18.273709 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273713 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:18.273907 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273721 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:18.273907 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273729 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:18.273907 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273736 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5wck\" (UniqueName: \"kubernetes.io/projected/208e7070-a527-4c0a-8ffc-7d459dbb5006-kube-api-access-n5wck\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:18.273907 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273744 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208e7070-a527-4c0a-8ffc-7d459dbb5006-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:18.273907 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273752 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:18.273907 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:18.273760 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208e7070-a527-4c0a-8ffc-7d459dbb5006-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:19.034289 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:19.034248 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" event={"ID":"208e7070-a527-4c0a-8ffc-7d459dbb5006","Type":"ContainerDied","Data":"728bb215384451de2d3ec442a9e3a6a76a4f8c609224c0d705da82f80e1f198a"} Jul 27 17:38:19.034723 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:19.034307 2557 scope.go:117] "RemoveContainer" containerID="cc7a33568e56fb3e6ae00e27abefc1fa5c048147f04423c1e6a35dd91e50ccfe" Jul 27 17:38:19.034723 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:19.034264 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q" Jul 27 17:38:19.056618 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:19.056590 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q"] Jul 27 17:38:19.060585 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:19.060560 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f6e958-openshift-default-d487bb446-ngj2q"] Jul 27 17:38:20.063766 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:20.063730 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" path="/var/lib/kubelet/pods/208e7070-a527-4c0a-8ffc-7d459dbb5006/volumes" Jul 27 17:38:28.824326 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:28.824290 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:28.910592 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:28.910554 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:28.969117 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:28.969075 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:33.450154 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:33.450117 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:33.642611 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:33.642578 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.036322 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:40.036244 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.102279 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:40.102246 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.289961 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:40.289866 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865"] Jul 27 17:38:40.290158 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:40.290111 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" containerID="cri-o://860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8" gracePeriod=30 Jul 27 17:38:40.325639 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:40.325605 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.529907 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:40.529874 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.865004 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:40.864964 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:41.149485 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.149447 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn"] Jul 27 17:38:41.149974 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.149954 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" Jul 27 17:38:41.150066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.149977 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" Jul 27 17:38:41.150066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.149989 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" Jul 27 17:38:41.150066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.149998 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" Jul 27 17:38:41.150066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.150024 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" Jul 27 17:38:41.150066 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.150033 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" Jul 27 17:38:41.150352 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.150117 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" Jul 27 17:38:41.150352 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.150130 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="208e7070-a527-4c0a-8ffc-7d459dbb5006" containerName="istio-proxy" Jul 27 17:38:41.150352 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.150142 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43041b3-5298-4ad9-bf5d-d3678702cd11" containerName="istio-proxy" Jul 27 17:38:41.152215 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.152171 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.158223 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.158191 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-004618-openshift-default-dockercfg-h9rlw\"" Jul 27 17:38:41.180504 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.180477 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn"] Jul 27 17:38:41.268130 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.268092 2557 patch_prober.go:28] interesting pod/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 17:38:41.268322 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.268146 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:38:41.288418 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288383 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-envoy\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.288418 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288429 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-istio-token\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.288692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-credential-socket\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.288692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288512 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-socket\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.288692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288546 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-data\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.288692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288568 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb83b28-897a-4917-8189-8740d286cf42-istiod-ca-cert\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.288692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmtf\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-kube-api-access-blmtf\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.288692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288623 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-certs\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.288692 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.288666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb83b28-897a-4917-8189-8740d286cf42-istio-podinfo\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.389690 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389647 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-envoy\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.389885 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389714 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-istio-token\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.389885 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-credential-socket\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.389885 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-socket\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.389885 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389822 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-data\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.389885 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389845 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb83b28-897a-4917-8189-8740d286cf42-istiod-ca-cert\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.389885 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389878 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmtf\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-kube-api-access-blmtf\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.390242 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-certs\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.390242 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.389949 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb83b28-897a-4917-8189-8740d286cf42-istio-podinfo\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.390242 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.390208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-data\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.390400 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.390237 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-socket\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.390449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.390411 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-credential-socket\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.390513 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.390469 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-certs\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.390689 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.390668 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb83b28-897a-4917-8189-8740d286cf42-istiod-ca-cert\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.391947 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.391927 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-envoy\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.392376 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.392360 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb83b28-897a-4917-8189-8740d286cf42-istio-podinfo\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.398306 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.398283 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-istio-token\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.398994 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.398975 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmtf\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-kube-api-access-blmtf\") pod \"e2e-shared-a-004618-openshift-default-6c499d8568-25gcn\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.462210 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.462121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:41.604941 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.604879 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn"] Jul 27 17:38:41.607071 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:38:41.607035 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb83b28_897a_4917_8189_8740d286cf42.slice/crio-81c046bf55abb57dd2c3e7d79352b326cd67b7e99c8eb534f8153d095cdf43bc WatchSource:0}: Error finding container 81c046bf55abb57dd2c3e7d79352b326cd67b7e99c8eb534f8153d095cdf43bc: Status 404 returned error can't find the container with id 81c046bf55abb57dd2c3e7d79352b326cd67b7e99c8eb534f8153d095cdf43bc Jul 27 17:38:41.609501 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.609464 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:38:41.609590 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.609547 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:38:41.609631 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.609591 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:38:41.734470 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.734383 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:41.878042 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:41.878011 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:42.138957 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:42.138924 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" event={"ID":"bcb83b28-897a-4917-8189-8740d286cf42","Type":"ContainerStarted","Data":"c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a"} Jul 27 17:38:42.139094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:42.138974 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" event={"ID":"bcb83b28-897a-4917-8189-8740d286cf42","Type":"ContainerStarted","Data":"81c046bf55abb57dd2c3e7d79352b326cd67b7e99c8eb534f8153d095cdf43bc"} Jul 27 17:38:42.161064 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:42.161015 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" podStartSLOduration=1.160999968 podStartE2EDuration="1.160999968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:42.160236948 +0000 UTC m=+2518.630563145" watchObservedRunningTime="2026-07-27 17:38:42.160999968 +0000 UTC m=+2518.631326163" Jul 27 17:38:42.463154 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:42.463073 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:42.468123 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:42.468098 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:43.144246 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:43.144209 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:43.145484 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:43.145457 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:38:44.034226 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:44.034159 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:44.131416 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:44.131367 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:44.230967 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:44.230930 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:44.335165 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:44.335082 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:44.750995 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:44.750964 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:44.850852 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:44.850816 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:45.547496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.547467 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:38:45.627794 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.627768 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-envoy\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.627981 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.627859 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-data\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.627981 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.627918 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-podinfo\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.627981 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.627940 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istiod-ca-cert\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.627981 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.627962 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-certs\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.627981 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.627980 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-credential-socket\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.628299 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628036 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-socket\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.628299 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628068 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfgrd\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-kube-api-access-rfgrd\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.628299 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628095 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-token\") pod \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\" (UID: \"7437bafa-405e-4c78-a0f9-bbf2fdf48aec\") " Jul 27 17:38:45.628443 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628336 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:45.628443 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628377 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:38:45.628619 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628452 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:45.628619 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628478 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:45.628852 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628827 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:45.628852 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.628838 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-data" (OuterVolumeSpecName: "istio-data") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:45.630144 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.630114 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:38:45.630144 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.630137 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:45.630354 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.630340 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-token" (OuterVolumeSpecName: "istio-token") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:38:45.630413 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.630395 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-kube-api-access-rfgrd" (OuterVolumeSpecName: "kube-api-access-rfgrd") pod "7437bafa-405e-4c78-a0f9-bbf2fdf48aec" (UID: "7437bafa-405e-4c78-a0f9-bbf2fdf48aec"). InnerVolumeSpecName "kube-api-access-rfgrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:38:45.728973 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.728937 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfgrd\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-kube-api-access-rfgrd\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:45.729143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.728968 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:45.729143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.729001 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:45.729143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.729010 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:45.729143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.729018 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:45.729143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.729026 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:45.729143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.729033 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:45.729143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:45.729041 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7437bafa-405e-4c78-a0f9-bbf2fdf48aec-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:38:46.158657 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.158623 2557 generic.go:356] "Generic (PLEG): container finished" podID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerID="860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8" exitCode=0 Jul 27 17:38:46.158860 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.158672 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" Jul 27 17:38:46.158860 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.158681 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" event={"ID":"7437bafa-405e-4c78-a0f9-bbf2fdf48aec","Type":"ContainerDied","Data":"860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8"} Jul 27 17:38:46.158860 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.158734 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865" event={"ID":"7437bafa-405e-4c78-a0f9-bbf2fdf48aec","Type":"ContainerDied","Data":"4e20eeb58dcbedefbab0f60905f3cc39227f0f8c781f74bc08acdf3dc0a776ad"} Jul 27 17:38:46.158860 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.158763 2557 scope.go:117] "RemoveContainer" containerID="860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8" Jul 27 17:38:46.167251 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.167230 2557 scope.go:117] "RemoveContainer" containerID="2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785" Jul 27 17:38:46.176271 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.176023 2557 scope.go:117] "RemoveContainer" containerID="860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8" Jul 27 17:38:46.178966 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:38:46.178527 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8\": container with ID starting with 860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8 not found: ID does not exist" containerID="860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8" Jul 27 17:38:46.178966 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.178570 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8"} err="failed to get container status \"860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8\": rpc error: code = NotFound desc = could not find container \"860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8\": container with ID starting with 860f59a5ad244f895902f73454c56c7aa7dd85b2076093fdcf44cdc81bcd53e8 not found: ID does not exist" Jul 27 17:38:46.178966 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.178598 2557 scope.go:117] "RemoveContainer" containerID="2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785" Jul 27 17:38:46.178966 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:38:46.178947 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785\": container with ID starting with 2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785 not found: ID does not exist" containerID="2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785" Jul 27 17:38:46.179272 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.178984 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785"} err="failed to get container status \"2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785\": rpc error: code = NotFound desc = could not find container \"2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785\": container with ID starting with 2903e363fd151e55fbe97753f9fb94ebee27e4678f6d3c76b362d4ca95eaa785 not found: ID does not exist" Jul 27 17:38:46.181266 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.181230 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865"] Jul 27 17:38:46.183142 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:46.183118 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-05e104-openshift-default-5ccdf7d7d4-9z865"] Jul 27 17:38:48.063850 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:48.063816 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" path="/var/lib/kubelet/pods/7437bafa-405e-4c78-a0f9-bbf2fdf48aec/volumes" Jul 27 17:38:48.504112 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:48.504083 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:48.843984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:48.843864 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:49.101896 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.101818 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7"] Jul 27 17:38:49.102285 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.102162 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" Jul 27 17:38:49.102285 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.102189 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" Jul 27 17:38:49.102285 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.102254 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" Jul 27 17:38:49.102285 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.102263 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" Jul 27 17:38:49.105367 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.105348 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.107453 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.107432 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-509a5d-openshift-default-dockercfg-h5x9g\"" Jul 27 17:38:49.117081 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.117042 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7"] Jul 27 17:38:49.153219 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.153186 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:49.157361 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157334 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-socket\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.157509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-credential-socket\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.157509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157399 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istiod-ca-cert\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.157509 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157448 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-certs\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.157694 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157510 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-envoy\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.157694 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157527 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-data\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.157694 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157554 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-podinfo\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.157694 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157584 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrp26\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-kube-api-access-mrp26\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.157694 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.157675 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-token\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.258852 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.258807 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-envoy\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.258852 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.258850 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-data\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.258867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-podinfo\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.258895 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrp26\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-kube-api-access-mrp26\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.258956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-token\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.258985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-socket\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.259028 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-credential-socket\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.259054 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istiod-ca-cert\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.259080 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-certs\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259457 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.259301 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-data\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259513 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.259453 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-credential-socket\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259513 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.259485 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-socket\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259589 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.259536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-certs\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.259833 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.259815 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istiod-ca-cert\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.261552 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.261528 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-envoy\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.261695 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.261676 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-podinfo\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.267644 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.267620 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-token\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.267785 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.267762 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrp26\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-kube-api-access-mrp26\") pod \"e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.394765 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.394734 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:49.416922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.416873 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:49.563106 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.563077 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7"] Jul 27 17:38:49.565100 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:38:49.565067 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04338905_db2a_4e0c_b599_ca4e2c5ba11c.slice/crio-6dfdd87ae9a552bde95092722d93d4d2105f9eb2318432c474796c20700891bb WatchSource:0}: Error finding container 6dfdd87ae9a552bde95092722d93d4d2105f9eb2318432c474796c20700891bb: Status 404 returned error can't find the container with id 6dfdd87ae9a552bde95092722d93d4d2105f9eb2318432c474796c20700891bb Jul 27 17:38:49.567314 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.567280 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:38:49.567408 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.567346 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:38:49.567408 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:49.567378 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:38:50.178678 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:50.178641 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" event={"ID":"04338905-db2a-4e0c-b599-ca4e2c5ba11c","Type":"ContainerStarted","Data":"bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b"} Jul 27 17:38:50.179062 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:50.178693 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" event={"ID":"04338905-db2a-4e0c-b599-ca4e2c5ba11c","Type":"ContainerStarted","Data":"6dfdd87ae9a552bde95092722d93d4d2105f9eb2318432c474796c20700891bb"} Jul 27 17:38:50.199556 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:50.199507 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" podStartSLOduration=1.199491551 podStartE2EDuration="1.199491551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:50.198675376 +0000 UTC m=+2526.669001582" watchObservedRunningTime="2026-07-27 17:38:50.199491551 +0000 UTC m=+2526.669817797" Jul 27 17:38:50.417617 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:50.417584 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:50.422500 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:50.422473 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:51.183115 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:51.183084 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:51.183990 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:51.183966 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:38:52.119192 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.119146 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:52.216733 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.216693 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:52.288987 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.288941 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:52.553558 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.553522 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:52.862389 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.862311 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:52.938445 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.938408 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn"] Jul 27 17:38:52.938795 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.938765 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" Jul 27 17:38:52.938881 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.938797 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7437bafa-405e-4c78-a0f9-bbf2fdf48aec" containerName="istio-proxy" Jul 27 17:38:52.941129 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.941102 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:52.941942 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.941924 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:52.944225 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.944202 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-509a5d-dockercfg-vgvtc\"" Jul 27 17:38:52.944335 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.944232 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-509a5d\"" Jul 27 17:38:52.951956 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:52.951932 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn"] Jul 27 17:38:53.095622 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:53.095581 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn\" (UID: \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:53.095790 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:53.095670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvs2\" (UniqueName: \"kubernetes.io/projected/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-kube-api-access-mlvs2\") pod \"payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn\" (UID: \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:53.196765 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:53.196729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvs2\" (UniqueName: \"kubernetes.io/projected/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-kube-api-access-mlvs2\") pod \"payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn\" (UID: \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:53.196953 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:53.196789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn\" (UID: \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:53.197475 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:53.197454 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn\" (UID: \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:53.204716 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:53.204689 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvs2\" (UniqueName: \"kubernetes.io/projected/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-kube-api-access-mlvs2\") pod \"payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn\" (UID: \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:53.255217 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:53.255161 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:53.430169 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:53.430140 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn"] Jul 27 17:38:53.432526 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:38:53.432497 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2aa048_5500_4cd1_9086_1f9a5e3e71af.slice/crio-37c1ca12deb8b18fe6ad8b120a1c0ff95c7814c17263662cb7b747162366fdb8 WatchSource:0}: Error finding container 37c1ca12deb8b18fe6ad8b120a1c0ff95c7814c17263662cb7b747162366fdb8: Status 404 returned error can't find the container with id 37c1ca12deb8b18fe6ad8b120a1c0ff95c7814c17263662cb7b747162366fdb8 Jul 27 17:38:54.198251 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:54.198213 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" event={"ID":"cb2aa048-5500-4cd1-9086-1f9a5e3e71af","Type":"ContainerStarted","Data":"72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae"} Jul 27 17:38:54.198436 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:54.198276 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" event={"ID":"cb2aa048-5500-4cd1-9086-1f9a5e3e71af","Type":"ContainerStarted","Data":"37c1ca12deb8b18fe6ad8b120a1c0ff95c7814c17263662cb7b747162366fdb8"} Jul 27 17:38:54.198436 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:54.198425 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:38:54.219026 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:54.218975 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" podStartSLOduration=2.218960153 podStartE2EDuration="2.218960153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:54.216780773 +0000 UTC m=+2530.687106969" watchObservedRunningTime="2026-07-27 17:38:54.218960153 +0000 UTC m=+2530.689286354" Jul 27 17:38:56.592903 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:56.592869 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:59.186420 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:59.186377 2557 patch_prober.go:28] interesting pod/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 17:38:59.186818 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:38:59.186451 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 17:39:01.233130 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:01.233096 2557 generic.go:356] "Generic (PLEG): container finished" podID="bcb83b28-897a-4917-8189-8740d286cf42" containerID="c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a" exitCode=0 Jul 27 17:39:01.233585 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:01.233169 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" event={"ID":"bcb83b28-897a-4917-8189-8740d286cf42","Type":"ContainerDied","Data":"c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a"} Jul 27 17:39:01.233585 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:01.233581 2557 scope.go:117] "RemoveContainer" containerID="c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a" Jul 27 17:39:01.234654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:01.234627 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:39:01.234711 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:01.234702 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:39:01.234755 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:01.234741 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:39:02.239917 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:02.239878 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" event={"ID":"bcb83b28-897a-4917-8189-8740d286cf42","Type":"ContainerStarted","Data":"5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99"} Jul 27 17:39:02.463005 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:02.462958 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:39:02.463493 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:02.463462 2557 patch_prober.go:28] interesting pod/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 27 17:39:02.463727 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:02.463699 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 27 17:39:03.463128 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:03.463089 2557 patch_prober.go:28] interesting pod/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 27 17:39:03.463753 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:03.463720 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 27 17:39:04.463650 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:04.463613 2557 patch_prober.go:28] interesting pod/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 27 17:39:04.464266 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:04.464224 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 27 17:39:05.205009 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:05.204974 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:39:05.463465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:05.463357 2557 patch_prober.go:28] interesting pod/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 27 17:39:05.463465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:05.463438 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 27 17:39:06.185095 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.185064 2557 patch_prober.go:28] interesting pod/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 27 17:39:06.185469 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.185121 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 27 17:39:06.285612 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.285580 2557 generic.go:356] "Generic (PLEG): container finished" podID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerID="bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b" exitCode=0 Jul 27 17:39:06.285782 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.285657 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" event={"ID":"04338905-db2a-4e0c-b599-ca4e2c5ba11c","Type":"ContainerDied","Data":"bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b"} Jul 27 17:39:06.286053 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.286039 2557 scope.go:117] "RemoveContainer" containerID="bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b" Jul 27 17:39:06.287014 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.286983 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:39:06.287120 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.287052 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:39:06.287120 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.287082 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:39:06.466778 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.466685 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:39:06.467079 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.467059 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:39:06.467954 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:06.467931 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:39:07.292488 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:07.292452 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" event={"ID":"04338905-db2a-4e0c-b599-ca4e2c5ba11c","Type":"ContainerStarted","Data":"d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5"} Jul 27 17:39:07.417496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:07.417457 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:39:07.417669 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:07.417654 2557 patch_prober.go:28] interesting pod/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 27 17:39:07.417713 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:07.417698 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 27 17:39:07.752891 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:07.752774 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:39:08.418302 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:08.418262 2557 patch_prober.go:28] interesting pod/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 27 17:39:08.418708 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:08.418324 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 27 17:39:09.418281 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:09.418242 2557 patch_prober.go:28] interesting pod/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 27 17:39:09.418470 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:09.418298 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 27 17:39:10.418146 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:10.418109 2557 patch_prober.go:28] interesting pod/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 27 17:39:10.418344 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:10.418165 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 27 17:39:11.422167 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:11.422134 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:39:11.422553 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:11.422401 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:39:11.423074 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:11.423056 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:39:34.164964 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:39:34.164920 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:19.566618 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:40:19.566507 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:19.671374 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:40:19.671313 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:36.126852 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:40:36.126801 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:44.226722 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:41:44.226643 2557 scope.go:117] "RemoveContainer" containerID="5eea61c3d70d8f25c089c38ee6a4fbea66c96a76a7d71564fffacc858119f521" Jul 27 17:42:29.891569 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:42:29.891535 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:43:00.348104 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:43:00.348047 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:43:12.396849 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:43:12.396767 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:43:12.456044 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:43:12.456013 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:43:49.908157 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:43:49.908113 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:43:50.647102 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:43:50.647067 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:43:53.081877 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:43:53.081842 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:43:53.184052 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:43:53.184016 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:43:53.283845 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:43:53.283815 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:44:12.836143 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:44:12.836110 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:45:45.607100 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:45:45.607063 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:45:46.455466 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:45:46.455413 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:46:00.561752 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:46:00.561717 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:46:28.131507 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:46:28.131421 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:47:06.796052 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:47:06.796016 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:47:42.967076 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:47:42.966985 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:47:45.148097 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:47:45.148059 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:03.613482 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:03.613439 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:04.067771 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:04.067741 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:04.355271 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:04.355167 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:06.925214 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:06.925158 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:07.020308 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:07.020266 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:07.142554 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:07.142518 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:07.284107 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:07.284011 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:07.982891 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:07.982847 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:08.082980 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.082948 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:08.198690 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.198653 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb"] Jul 27 17:48:08.201924 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.201904 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.205897 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.205872 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d401e5-dockercfg-pqlv8\"" Jul 27 17:48:08.206290 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.206274 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d401e5\"" Jul 27 17:48:08.238996 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.238906 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb"] Jul 27 17:48:08.291101 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.291068 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgw4g\" (UniqueName: \"kubernetes.io/projected/ce55b173-3f4e-4782-a714-eece9ad9d566-kube-api-access-pgw4g\") pod \"payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb\" (UID: \"ce55b173-3f4e-4782-a714-eece9ad9d566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.291297 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.291130 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce55b173-3f4e-4782-a714-eece9ad9d566-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb\" (UID: \"ce55b173-3f4e-4782-a714-eece9ad9d566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.392148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.392112 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce55b173-3f4e-4782-a714-eece9ad9d566-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb\" (UID: \"ce55b173-3f4e-4782-a714-eece9ad9d566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.392331 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.392236 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgw4g\" (UniqueName: \"kubernetes.io/projected/ce55b173-3f4e-4782-a714-eece9ad9d566-kube-api-access-pgw4g\") pod \"payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb\" (UID: \"ce55b173-3f4e-4782-a714-eece9ad9d566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.392856 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.392836 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce55b173-3f4e-4782-a714-eece9ad9d566-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb\" (UID: \"ce55b173-3f4e-4782-a714-eece9ad9d566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.406617 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.406582 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgw4g\" (UniqueName: \"kubernetes.io/projected/ce55b173-3f4e-4782-a714-eece9ad9d566-kube-api-access-pgw4g\") pod \"payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb\" (UID: \"ce55b173-3f4e-4782-a714-eece9ad9d566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.512613 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.512521 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.660532 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.660508 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb"] Jul 27 17:48:08.662506 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:48:08.662479 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce55b173_3f4e_4782_a714_eece9ad9d566.slice/crio-5fa55f8bda40e21ccda00293f2a02e760e2d86331cf964f65045d670b78b8457 WatchSource:0}: Error finding container 5fa55f8bda40e21ccda00293f2a02e760e2d86331cf964f65045d670b78b8457: Status 404 returned error can't find the container with id 5fa55f8bda40e21ccda00293f2a02e760e2d86331cf964f65045d670b78b8457 Jul 27 17:48:08.852560 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.852467 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" event={"ID":"ce55b173-3f4e-4782-a714-eece9ad9d566","Type":"ContainerStarted","Data":"edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304"} Jul 27 17:48:08.852560 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.852546 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:08.852793 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.852571 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" event={"ID":"ce55b173-3f4e-4782-a714-eece9ad9d566","Type":"ContainerStarted","Data":"5fa55f8bda40e21ccda00293f2a02e760e2d86331cf964f65045d670b78b8457"} Jul 27 17:48:08.878248 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:08.878167 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" podStartSLOduration=0.878148491 podStartE2EDuration="878.148491ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:08.877812555 +0000 UTC m=+3085.348138784" watchObservedRunningTime="2026-07-27 17:48:08.878148491 +0000 UTC m=+3085.348474687" Jul 27 17:48:11.380026 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:11.379986 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:19.859276 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:19.859243 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:48:22.294035 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.294001 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:22.606458 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.606380 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg"] Jul 27 17:48:22.610217 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.610195 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.613706 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.613427 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3bb3ae-openshift-default-dockercfg-nngws\"" Jul 27 17:48:22.623749 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.623726 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg"] Jul 27 17:48:22.648562 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.648529 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:22.721793 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.721748 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-socket\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.721984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.721799 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-token\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.721984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.721849 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-data\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.721984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.721873 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf380c91-83f8-44e8-be87-50eff59bfc6c-istiod-ca-cert\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.721984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.721898 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-podinfo\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.721984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.721933 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-credential-socket\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.721984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.721956 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-certs\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.721984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.721978 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlk29\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-kube-api-access-jlk29\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.722382 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.722043 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-envoy\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823222 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-socket\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823415 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-token\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823415 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823275 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-data\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823415 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf380c91-83f8-44e8-be87-50eff59bfc6c-istiod-ca-cert\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823415 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823325 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-podinfo\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823415 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823362 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-credential-socket\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823415 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823387 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-certs\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823415 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823411 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlk29\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-kube-api-access-jlk29\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823772 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823478 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-envoy\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823772 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823629 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-socket\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823772 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823679 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-data\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.823923 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.823791 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-credential-socket\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.824063 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.824040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf380c91-83f8-44e8-be87-50eff59bfc6c-istiod-ca-cert\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.824122 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.824049 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-certs\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.825789 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.825767 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-envoy\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.825922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.825908 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-podinfo\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.831595 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.831569 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-token\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.831812 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.831791 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlk29\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-kube-api-access-jlk29\") pod \"e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:22.921294 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:22.921267 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:23.068539 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:23.068512 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg"] Jul 27 17:48:23.070063 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:48:23.070036 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf380c91_83f8_44e8_be87_50eff59bfc6c.slice/crio-a8308c16a5c0f524f18265424009cda5bfef1c94141a179fad491369ecb2d6b4 WatchSource:0}: Error finding container a8308c16a5c0f524f18265424009cda5bfef1c94141a179fad491369ecb2d6b4: Status 404 returned error can't find the container with id a8308c16a5c0f524f18265424009cda5bfef1c94141a179fad491369ecb2d6b4 Jul 27 17:48:23.072582 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:23.072541 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:48:23.072683 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:23.072643 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:48:23.072735 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:23.072686 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:48:23.102346 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:23.102314 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:23.926954 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:23.926917 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" event={"ID":"cf380c91-83f8-44e8-be87-50eff59bfc6c","Type":"ContainerStarted","Data":"39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464"} Jul 27 17:48:23.927364 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:23.926976 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" event={"ID":"cf380c91-83f8-44e8-be87-50eff59bfc6c","Type":"ContainerStarted","Data":"a8308c16a5c0f524f18265424009cda5bfef1c94141a179fad491369ecb2d6b4"} Jul 27 17:48:23.954064 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:23.954013 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" podStartSLOduration=1.95399357 podStartE2EDuration="1.95399357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:23.953425681 +0000 UTC m=+3100.423751877" watchObservedRunningTime="2026-07-27 17:48:23.95399357 +0000 UTC m=+3100.424319764" Jul 27 17:48:24.922195 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:24.922136 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:24.927217 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:24.927185 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:24.931284 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:24.931262 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:24.931929 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:24.931910 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:25.898429 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:25.898379 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:25.980268 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:25.980237 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:26.047864 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:26.047827 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:26.194723 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:26.194621 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:26.951671 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:26.951637 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:27.051304 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.051266 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:27.083222 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.083190 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q"] Jul 27 17:48:27.088194 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.087840 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.090595 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.090569 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3bb3ae\"" Jul 27 17:48:27.090769 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.090629 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3bb3ae-dockercfg-gfz84\"" Jul 27 17:48:27.102064 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.102038 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q"] Jul 27 17:48:27.166542 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.166502 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10ad513-25ca-4881-be86-b659e4ce74d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q\" (UID: \"a10ad513-25ca-4881-be86-b659e4ce74d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.166737 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.166556 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2br\" (UniqueName: \"kubernetes.io/projected/a10ad513-25ca-4881-be86-b659e4ce74d1-kube-api-access-nc2br\") pod \"payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q\" (UID: \"a10ad513-25ca-4881-be86-b659e4ce74d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.267437 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.267352 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10ad513-25ca-4881-be86-b659e4ce74d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q\" (UID: \"a10ad513-25ca-4881-be86-b659e4ce74d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.267437 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.267424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2br\" (UniqueName: \"kubernetes.io/projected/a10ad513-25ca-4881-be86-b659e4ce74d1-kube-api-access-nc2br\") pod \"payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q\" (UID: \"a10ad513-25ca-4881-be86-b659e4ce74d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.268148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.268129 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10ad513-25ca-4881-be86-b659e4ce74d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q\" (UID: \"a10ad513-25ca-4881-be86-b659e4ce74d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.276740 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.276714 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2br\" (UniqueName: \"kubernetes.io/projected/a10ad513-25ca-4881-be86-b659e4ce74d1-kube-api-access-nc2br\") pod \"payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q\" (UID: \"a10ad513-25ca-4881-be86-b659e4ce74d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.399502 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.399467 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.546780 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.546753 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q"] Jul 27 17:48:27.548593 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:48:27.548559 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10ad513_25ca_4881_be86_b659e4ce74d1.slice/crio-67ce9575c541efc9d42006d8844569a62e908ddf4b5e60a2544bec47c15f8630 WatchSource:0}: Error finding container 67ce9575c541efc9d42006d8844569a62e908ddf4b5e60a2544bec47c15f8630: Status 404 returned error can't find the container with id 67ce9575c541efc9d42006d8844569a62e908ddf4b5e60a2544bec47c15f8630 Jul 27 17:48:27.946496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.946462 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" event={"ID":"a10ad513-25ca-4881-be86-b659e4ce74d1","Type":"ContainerStarted","Data":"09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d"} Jul 27 17:48:27.946709 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.946523 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" event={"ID":"a10ad513-25ca-4881-be86-b659e4ce74d1","Type":"ContainerStarted","Data":"67ce9575c541efc9d42006d8844569a62e908ddf4b5e60a2544bec47c15f8630"} Jul 27 17:48:27.946709 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.946575 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:27.968026 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:27.967973 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" podStartSLOduration=0.967958042 podStartE2EDuration="967.958042ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:27.966704119 +0000 UTC m=+3104.437030315" watchObservedRunningTime="2026-07-27 17:48:27.967958042 +0000 UTC m=+3104.438284236" Jul 27 17:48:30.361760 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:30.361722 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:38.955394 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:38.955361 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:48:40.973384 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:40.973335 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 17:48:40.973899 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:40.973416 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 17:48:43.023233 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:43.023129 2557 generic.go:356] "Generic (PLEG): container finished" podID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerID="39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464" exitCode=0 Jul 27 17:48:43.023233 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:43.023205 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" event={"ID":"cf380c91-83f8-44e8-be87-50eff59bfc6c","Type":"ContainerDied","Data":"39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464"} Jul 27 17:48:43.023762 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:43.023664 2557 scope.go:117] "RemoveContainer" containerID="39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464" Jul 27 17:48:43.024722 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:43.024691 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:48:43.024835 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:43.024777 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:48:43.024835 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:43.024828 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 17:48:44.029874 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:44.029834 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" event={"ID":"cf380c91-83f8-44e8-be87-50eff59bfc6c","Type":"ContainerStarted","Data":"b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2"} Jul 27 17:48:44.921728 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:44.921686 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:44.921938 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:44.921914 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 27 17:48:44.922000 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:44.921976 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 27 17:48:45.922553 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:45.922518 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 27 17:48:45.922952 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:45.922584 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 27 17:48:46.922426 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:46.922391 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 27 17:48:46.922602 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:46.922459 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 27 17:48:47.360985 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:47.360886 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vcgrx_69dfa449-4401-4cf9-8374-0f08e94eeeae/payload-processing/0.log" Jul 27 17:48:47.482605 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:47.482571 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q_a10ad513-25ca-4881-be86-b659e4ce74d1/payload-processing/0.log" Jul 27 17:48:47.924918 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:47.924885 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:47.925338 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:47.925219 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:47.925941 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:47.925920 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:48:54.852403 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:54.852352 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:55.536732 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:48:55.536699 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:39.721530 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:39.721446 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vcgrx_69dfa449-4401-4cf9-8374-0f08e94eeeae/payload-processing/0.log" Jul 27 17:49:39.855094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:39.855069 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q_a10ad513-25ca-4881-be86-b659e4ce74d1/payload-processing/0.log" Jul 27 17:49:40.151923 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:40.151891 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:40.443716 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:40.443629 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:40.800430 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:40.800347 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:43.460026 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:43.459994 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:43.558047 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:43.558011 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:43.669113 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:43.669074 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:43.977038 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:43.976986 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:44.322253 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.322122 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:44.425193 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.425134 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:44.442768 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.442735 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft"] Jul 27 17:49:44.446394 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.446369 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:44.449472 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.449442 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-4df9d8\"" Jul 27 17:49:44.459107 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.459081 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft"] Jul 27 17:49:44.478383 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.478356 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5427s\" (UniqueName: \"kubernetes.io/projected/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-kube-api-access-5427s\") pod \"payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft\" (UID: \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:44.478741 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.478500 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft\" (UID: \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:44.585256 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.582601 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft\" (UID: \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:44.585256 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.582716 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5427s\" (UniqueName: \"kubernetes.io/projected/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-kube-api-access-5427s\") pod \"payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft\" (UID: \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:44.585256 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.583628 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft\" (UID: \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:44.593580 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.593533 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5427s\" (UniqueName: \"kubernetes.io/projected/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-kube-api-access-5427s\") pod \"payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft\" (UID: \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:44.757870 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:44.757836 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:45.105410 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:45.105381 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft"] Jul 27 17:49:45.108282 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:49:45.108256 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3623ccbb_f221_4f54_b4a0_f0363b0cb97a.slice/crio-194c8b1f0e4c91f139f46be435a41810f2a89dc92c9409f7f37094173e04f3bb WatchSource:0}: Error finding container 194c8b1f0e4c91f139f46be435a41810f2a89dc92c9409f7f37094173e04f3bb: Status 404 returned error can't find the container with id 194c8b1f0e4c91f139f46be435a41810f2a89dc92c9409f7f37094173e04f3bb Jul 27 17:49:45.332083 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:45.332035 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" event={"ID":"3623ccbb-f221-4f54-b4a0-f0363b0cb97a","Type":"ContainerStarted","Data":"0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793"} Jul 27 17:49:45.332310 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:45.332097 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" event={"ID":"3623ccbb-f221-4f54-b4a0-f0363b0cb97a","Type":"ContainerStarted","Data":"194c8b1f0e4c91f139f46be435a41810f2a89dc92c9409f7f37094173e04f3bb"} Jul 27 17:49:45.332310 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:45.332128 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:47.976281 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:47.976250 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:56.339927 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:56.339891 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:49:56.365643 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:49:56.365593 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" podStartSLOduration=12.365578424 podStartE2EDuration="12.365578424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:49:45.351484686 +0000 UTC m=+3181.821810883" watchObservedRunningTime="2026-07-27 17:49:56.365578424 +0000 UTC m=+3192.835904619" Jul 27 17:50:09.658227 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:09.658157 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:09.706615 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:09.706580 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft"] Jul 27 17:50:09.706851 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:09.706823 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" podUID="3623ccbb-f221-4f54-b4a0-f0363b0cb97a" containerName="payload-pre-processing" containerID="cri-o://0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793" gracePeriod=30 Jul 27 17:50:09.736600 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:09.736563 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:09.813241 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:09.813161 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:09.982994 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:09.982968 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:50:10.138098 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.138068 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-plugins-config-volume\") pod \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\" (UID: \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\") " Jul 27 17:50:10.138318 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.138109 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5427s\" (UniqueName: \"kubernetes.io/projected/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-kube-api-access-5427s\") pod \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\" (UID: \"3623ccbb-f221-4f54-b4a0-f0363b0cb97a\") " Jul 27 17:50:10.138501 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.138475 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3623ccbb-f221-4f54-b4a0-f0363b0cb97a" (UID: "3623ccbb-f221-4f54-b4a0-f0363b0cb97a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:50:10.140310 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.140287 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-kube-api-access-5427s" (OuterVolumeSpecName: "kube-api-access-5427s") pod "3623ccbb-f221-4f54-b4a0-f0363b0cb97a" (UID: "3623ccbb-f221-4f54-b4a0-f0363b0cb97a"). InnerVolumeSpecName "kube-api-access-5427s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:50:10.239383 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.239297 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:50:10.239383 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.239326 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5427s\" (UniqueName: \"kubernetes.io/projected/3623ccbb-f221-4f54-b4a0-f0363b0cb97a-kube-api-access-5427s\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:50:10.453666 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.453627 2557 generic.go:356] "Generic (PLEG): container finished" podID="3623ccbb-f221-4f54-b4a0-f0363b0cb97a" containerID="0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793" exitCode=0 Jul 27 17:50:10.453849 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.453684 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" Jul 27 17:50:10.453849 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.453709 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" event={"ID":"3623ccbb-f221-4f54-b4a0-f0363b0cb97a","Type":"ContainerDied","Data":"0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793"} Jul 27 17:50:10.453849 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.453781 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft" event={"ID":"3623ccbb-f221-4f54-b4a0-f0363b0cb97a","Type":"ContainerDied","Data":"194c8b1f0e4c91f139f46be435a41810f2a89dc92c9409f7f37094173e04f3bb"} Jul 27 17:50:10.453849 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.453804 2557 scope.go:117] "RemoveContainer" containerID="0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793" Jul 27 17:50:10.462795 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.462775 2557 scope.go:117] "RemoveContainer" containerID="0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793" Jul 27 17:50:10.463058 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:50:10.463038 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793\": container with ID starting with 0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793 not found: ID does not exist" containerID="0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793" Jul 27 17:50:10.463133 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.463066 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793"} err="failed to get container status \"0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793\": rpc error: code = NotFound desc = could not find container \"0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793\": container with ID starting with 0532d9f7b6903e537e17f95f33e77205aa1003cd43075de6c112c26c25e1a793 not found: ID does not exist" Jul 27 17:50:10.476287 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.476266 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft"] Jul 27 17:50:10.480946 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:10.480918 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4df9d8-6c76cff44-cqhft"] Jul 27 17:50:12.064040 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:12.064005 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3623ccbb-f221-4f54-b4a0-f0363b0cb97a" path="/var/lib/kubelet/pods/3623ccbb-f221-4f54-b4a0-f0363b0cb97a/volumes" Jul 27 17:50:14.257137 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:14.257100 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:14.477755 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:14.477723 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:20.350367 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:20.350329 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:20.586958 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:20.586925 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:20.708835 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:20.708799 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:20.952040 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:20.952003 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:40.233045 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.232960 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:40.319067 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.319038 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb"] Jul 27 17:50:40.319265 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.319242 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" podUID="ce55b173-3f4e-4782-a714-eece9ad9d566" containerName="payload-pre-processing" containerID="cri-o://edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304" gracePeriod=30 Jul 27 17:50:40.356444 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.356374 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:40.452021 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.451980 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:40.596350 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.596326 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:50:40.603444 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.603411 2557 generic.go:356] "Generic (PLEG): container finished" podID="ce55b173-3f4e-4782-a714-eece9ad9d566" containerID="edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304" exitCode=0 Jul 27 17:50:40.603588 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.603524 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" Jul 27 17:50:40.603588 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.603555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" event={"ID":"ce55b173-3f4e-4782-a714-eece9ad9d566","Type":"ContainerDied","Data":"edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304"} Jul 27 17:50:40.603700 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.603604 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb" event={"ID":"ce55b173-3f4e-4782-a714-eece9ad9d566","Type":"ContainerDied","Data":"5fa55f8bda40e21ccda00293f2a02e760e2d86331cf964f65045d670b78b8457"} Jul 27 17:50:40.603700 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.603632 2557 scope.go:117] "RemoveContainer" containerID="edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304" Jul 27 17:50:40.613260 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.613239 2557 scope.go:117] "RemoveContainer" containerID="edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304" Jul 27 17:50:40.613541 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:50:40.613520 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304\": container with ID starting with edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304 not found: ID does not exist" containerID="edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304" Jul 27 17:50:40.613688 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.613554 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304"} err="failed to get container status \"edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304\": rpc error: code = NotFound desc = could not find container \"edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304\": container with ID starting with edaa5536c9c310e424de2b290940cdd67ce3c7974ffc187fe13477b8c9e2e304 not found: ID does not exist" Jul 27 17:50:40.729287 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.729252 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgw4g\" (UniqueName: \"kubernetes.io/projected/ce55b173-3f4e-4782-a714-eece9ad9d566-kube-api-access-pgw4g\") pod \"ce55b173-3f4e-4782-a714-eece9ad9d566\" (UID: \"ce55b173-3f4e-4782-a714-eece9ad9d566\") " Jul 27 17:50:40.729287 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.729290 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce55b173-3f4e-4782-a714-eece9ad9d566-plugins-config-volume\") pod \"ce55b173-3f4e-4782-a714-eece9ad9d566\" (UID: \"ce55b173-3f4e-4782-a714-eece9ad9d566\") " Jul 27 17:50:40.729710 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.729685 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce55b173-3f4e-4782-a714-eece9ad9d566-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce55b173-3f4e-4782-a714-eece9ad9d566" (UID: "ce55b173-3f4e-4782-a714-eece9ad9d566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:50:40.731449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.731425 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce55b173-3f4e-4782-a714-eece9ad9d566-kube-api-access-pgw4g" (OuterVolumeSpecName: "kube-api-access-pgw4g") pod "ce55b173-3f4e-4782-a714-eece9ad9d566" (UID: "ce55b173-3f4e-4782-a714-eece9ad9d566"). InnerVolumeSpecName "kube-api-access-pgw4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:50:40.830558 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.830466 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgw4g\" (UniqueName: \"kubernetes.io/projected/ce55b173-3f4e-4782-a714-eece9ad9d566-kube-api-access-pgw4g\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:50:40.830558 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.830500 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce55b173-3f4e-4782-a714-eece9ad9d566-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:50:40.927100 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.927065 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb"] Jul 27 17:50:40.930973 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:40.930943 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d401e5-6dfff5675b-74pqb"] Jul 27 17:50:42.064819 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:42.064786 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce55b173-3f4e-4782-a714-eece9ad9d566" path="/var/lib/kubelet/pods/ce55b173-3f4e-4782-a714-eece9ad9d566/volumes" Jul 27 17:50:44.783860 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:44.783814 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:45.153853 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:45.153819 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:50.223370 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:50:50.223333 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:16.359999 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:16.359943 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:27.028228 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:27.028188 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:27.863518 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:27.863472 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:34.576290 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:34.576244 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:39.616371 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:39.616327 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:39.678628 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:39.678578 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:39.904192 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:39.904146 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:40.137987 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:40.137952 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:50.900763 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:50.900722 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:50.947106 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:50.947073 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q"] Jul 27 17:51:50.947650 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:50.947348 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" podUID="a10ad513-25ca-4881-be86-b659e4ce74d1" containerName="payload-processing" containerID="cri-o://09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d" gracePeriod=30 Jul 27 17:51:51.003412 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.003375 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:51.214873 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.214845 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:51:51.281302 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.281273 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc2br\" (UniqueName: \"kubernetes.io/projected/a10ad513-25ca-4881-be86-b659e4ce74d1-kube-api-access-nc2br\") pod \"a10ad513-25ca-4881-be86-b659e4ce74d1\" (UID: \"a10ad513-25ca-4881-be86-b659e4ce74d1\") " Jul 27 17:51:51.281484 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.281465 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10ad513-25ca-4881-be86-b659e4ce74d1-plugins-config-volume\") pod \"a10ad513-25ca-4881-be86-b659e4ce74d1\" (UID: \"a10ad513-25ca-4881-be86-b659e4ce74d1\") " Jul 27 17:51:51.281823 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.281799 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10ad513-25ca-4881-be86-b659e4ce74d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a10ad513-25ca-4881-be86-b659e4ce74d1" (UID: "a10ad513-25ca-4881-be86-b659e4ce74d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:51:51.283340 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.283317 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10ad513-25ca-4881-be86-b659e4ce74d1-kube-api-access-nc2br" (OuterVolumeSpecName: "kube-api-access-nc2br") pod "a10ad513-25ca-4881-be86-b659e4ce74d1" (UID: "a10ad513-25ca-4881-be86-b659e4ce74d1"). InnerVolumeSpecName "kube-api-access-nc2br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:51:51.382148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.382113 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10ad513-25ca-4881-be86-b659e4ce74d1-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:51:51.382148 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.382143 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc2br\" (UniqueName: \"kubernetes.io/projected/a10ad513-25ca-4881-be86-b659e4ce74d1-kube-api-access-nc2br\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:51:51.382867 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.382831 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:51.956662 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.956627 2557 generic.go:356] "Generic (PLEG): container finished" podID="a10ad513-25ca-4881-be86-b659e4ce74d1" containerID="09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d" exitCode=0 Jul 27 17:51:51.957081 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.956672 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" Jul 27 17:51:51.957081 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.956714 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" event={"ID":"a10ad513-25ca-4881-be86-b659e4ce74d1","Type":"ContainerDied","Data":"09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d"} Jul 27 17:51:51.957081 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.956764 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q" event={"ID":"a10ad513-25ca-4881-be86-b659e4ce74d1","Type":"ContainerDied","Data":"67ce9575c541efc9d42006d8844569a62e908ddf4b5e60a2544bec47c15f8630"} Jul 27 17:51:51.957081 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.956786 2557 scope.go:117] "RemoveContainer" containerID="09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d" Jul 27 17:51:51.965816 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.965800 2557 scope.go:117] "RemoveContainer" containerID="09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d" Jul 27 17:51:51.966068 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:51:51.966048 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d\": container with ID starting with 09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d not found: ID does not exist" containerID="09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d" Jul 27 17:51:51.966130 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.966081 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d"} err="failed to get container status \"09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d\": rpc error: code = NotFound desc = could not find container \"09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d\": container with ID starting with 09c6d0a88800755500e3f3fe93d5345bc006414d20ce49fe64e411f528fa5d6d not found: ID does not exist" Jul 27 17:51:51.978096 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.978073 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q"] Jul 27 17:51:51.982158 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:51.982134 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3bb3ae-7cf746bcc6-p8b2q"] Jul 27 17:51:52.064189 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:52.064154 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10ad513-25ca-4881-be86-b659e4ce74d1" path="/var/lib/kubelet/pods/a10ad513-25ca-4881-be86-b659e4ce74d1/volumes" Jul 27 17:51:53.323365 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:53.323309 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:55.363302 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:55.363266 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:55.742769 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:51:55.742725 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:02.204013 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:02.203953 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:02.269257 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:02.269220 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:02.464577 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:02.464476 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg"] Jul 27 17:52:02.464750 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:02.464715 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" containerID="cri-o://b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2" gracePeriod=30 Jul 27 17:52:02.482250 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:02.482220 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:02.706332 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:02.706297 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:02.927672 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:02.927631 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 17:52:02.927829 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:02.927681 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:52:07.723924 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.723896 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:52:07.830868 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.830778 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-podinfo\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.830868 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.830816 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlk29\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-kube-api-access-jlk29\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.830868 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.830855 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-credential-socket\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.831157 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.830877 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-certs\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.831157 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831010 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf380c91-83f8-44e8-be87-50eff59bfc6c-istiod-ca-cert\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.831157 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831062 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-data\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.831157 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831137 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-token\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.831395 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831159 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:07.831395 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831213 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-envoy\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.831395 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831285 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-socket\") pod \"cf380c91-83f8-44e8-be87-50eff59bfc6c\" (UID: \"cf380c91-83f8-44e8-be87-50eff59bfc6c\") " Jul 27 17:52:07.831395 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831283 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:07.831601 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831419 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf380c91-83f8-44e8-be87-50eff59bfc6c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:52:07.831601 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831529 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:07.831878 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831861 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:07.831962 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831881 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:07.831962 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831895 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf380c91-83f8-44e8-be87-50eff59bfc6c-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:07.831962 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.831907 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:07.832087 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.832056 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:07.833164 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.833134 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:52:07.833280 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.833193 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:07.833474 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.833455 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:52:07.833518 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.833483 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-kube-api-access-jlk29" (OuterVolumeSpecName: "kube-api-access-jlk29") pod "cf380c91-83f8-44e8-be87-50eff59bfc6c" (UID: "cf380c91-83f8-44e8-be87-50eff59bfc6c"). InnerVolumeSpecName "kube-api-access-jlk29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:52:07.933125 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.933090 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:07.933125 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.933120 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlk29\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-kube-api-access-jlk29\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:07.933125 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.933130 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:07.933378 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.933139 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:07.933378 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:07.933146 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf380c91-83f8-44e8-be87-50eff59bfc6c-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:08.038793 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.038756 2557 generic.go:356] "Generic (PLEG): container finished" podID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerID="b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2" exitCode=0 Jul 27 17:52:08.038958 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.038799 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" event={"ID":"cf380c91-83f8-44e8-be87-50eff59bfc6c","Type":"ContainerDied","Data":"b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2"} Jul 27 17:52:08.038958 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.038811 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" Jul 27 17:52:08.038958 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.038875 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg" event={"ID":"cf380c91-83f8-44e8-be87-50eff59bfc6c","Type":"ContainerDied","Data":"a8308c16a5c0f524f18265424009cda5bfef1c94141a179fad491369ecb2d6b4"} Jul 27 17:52:08.038958 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.038900 2557 scope.go:117] "RemoveContainer" containerID="b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2" Jul 27 17:52:08.047960 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.047942 2557 scope.go:117] "RemoveContainer" containerID="39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464" Jul 27 17:52:08.055758 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.055741 2557 scope.go:117] "RemoveContainer" containerID="b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2" Jul 27 17:52:08.056006 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:52:08.055984 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2\": container with ID starting with b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2 not found: ID does not exist" containerID="b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2" Jul 27 17:52:08.056078 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.056014 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2"} err="failed to get container status \"b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2\": rpc error: code = NotFound desc = could not find container \"b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2\": container with ID starting with b1110157e56280cab040e33f426c01a31ca3f444f5db14417c068b6b81a519e2 not found: ID does not exist" Jul 27 17:52:08.056078 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.056031 2557 scope.go:117] "RemoveContainer" containerID="39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464" Jul 27 17:52:08.056275 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:52:08.056253 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464\": container with ID starting with 39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464 not found: ID does not exist" containerID="39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464" Jul 27 17:52:08.056368 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.056279 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464"} err="failed to get container status \"39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464\": rpc error: code = NotFound desc = could not find container \"39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464\": container with ID starting with 39b9f071b23ce8ff4a6d31bf67fc081fc46d75f4b1a5c6c51cd68271332aa464 not found: ID does not exist" Jul 27 17:52:08.064415 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.064394 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg"] Jul 27 17:52:08.067297 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:08.067275 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3bb3ae-openshift-default-745fdc5b-v9szg"] Jul 27 17:52:10.063994 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:10.063959 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" path="/var/lib/kubelet/pods/cf380c91-83f8-44e8-be87-50eff59bfc6c/volumes" Jul 27 17:52:11.709317 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:11.709271 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:13.618496 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:13.618453 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:13.719340 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:13.719294 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:13.816930 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:13.816880 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:14.327687 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:14.327655 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:14.404351 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:14.404311 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:18.184492 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:18.184453 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:18.324407 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:18.324376 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:24.892525 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:24.892492 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:25.153001 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:25.152906 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn"] Jul 27 17:52:25.153257 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:25.153210 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" containerID="cri-o://5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99" gracePeriod=30 Jul 27 17:52:25.486730 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:25.486638 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:25.554693 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:25.554655 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:30.412004 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.411940 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:52:30.433962 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.433932 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-data\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434029 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-istio-token\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434047 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-socket\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434070 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb83b28-897a-4917-8189-8740d286cf42-istiod-ca-cert\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434091 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434091 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb83b28-897a-4917-8189-8740d286cf42-istio-podinfo\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434120 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-envoy\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434153 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blmtf\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-kube-api-access-blmtf\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434210 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-certs\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434254 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-credential-socket\") pod \"bcb83b28-897a-4917-8189-8740d286cf42\" (UID: \"bcb83b28-897a-4917-8189-8740d286cf42\") " Jul 27 17:52:30.434427 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434331 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:30.434558 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434527 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:30.435099 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434638 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb83b28-897a-4917-8189-8740d286cf42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:52:30.435099 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434799 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-data" (OuterVolumeSpecName: "istio-data") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:30.435099 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.434998 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:30.436003 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.435239 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:30.437751 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.437713 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:30.437849 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.437794 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-istio-token" (OuterVolumeSpecName: "istio-token") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:52:30.437920 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.437869 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcb83b28-897a-4917-8189-8740d286cf42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:52:30.438135 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.438113 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-kube-api-access-blmtf" (OuterVolumeSpecName: "kube-api-access-blmtf") pod "bcb83b28-897a-4917-8189-8740d286cf42" (UID: "bcb83b28-897a-4917-8189-8740d286cf42"). InnerVolumeSpecName "kube-api-access-blmtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:52:30.535665 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.535629 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:30.535665 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.535660 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:30.535665 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.535669 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb83b28-897a-4917-8189-8740d286cf42-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:30.535897 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.535677 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb83b28-897a-4917-8189-8740d286cf42-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:30.535897 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.535685 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:30.535897 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.535694 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmtf\" (UniqueName: \"kubernetes.io/projected/bcb83b28-897a-4917-8189-8740d286cf42-kube-api-access-blmtf\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:30.535897 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.535704 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:30.535897 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:30.535712 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb83b28-897a-4917-8189-8740d286cf42-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:31.144262 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.144226 2557 generic.go:356] "Generic (PLEG): container finished" podID="bcb83b28-897a-4917-8189-8740d286cf42" containerID="5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99" exitCode=0 Jul 27 17:52:31.144465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.144280 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" Jul 27 17:52:31.144465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.144302 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" event={"ID":"bcb83b28-897a-4917-8189-8740d286cf42","Type":"ContainerDied","Data":"5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99"} Jul 27 17:52:31.144465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.144348 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn" event={"ID":"bcb83b28-897a-4917-8189-8740d286cf42","Type":"ContainerDied","Data":"81c046bf55abb57dd2c3e7d79352b326cd67b7e99c8eb534f8153d095cdf43bc"} Jul 27 17:52:31.144465 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.144372 2557 scope.go:117] "RemoveContainer" containerID="5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99" Jul 27 17:52:31.153669 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.153651 2557 scope.go:117] "RemoveContainer" containerID="c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a" Jul 27 17:52:31.161664 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.161649 2557 scope.go:117] "RemoveContainer" containerID="5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99" Jul 27 17:52:31.161903 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:52:31.161883 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99\": container with ID starting with 5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99 not found: ID does not exist" containerID="5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99" Jul 27 17:52:31.161945 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.161912 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99"} err="failed to get container status \"5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99\": rpc error: code = NotFound desc = could not find container \"5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99\": container with ID starting with 5ec32824cb122370075638775556175470cd0aa665a4573bec0bebe315a67c99 not found: ID does not exist" Jul 27 17:52:31.161945 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.161930 2557 scope.go:117] "RemoveContainer" containerID="c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a" Jul 27 17:52:31.162136 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:52:31.162122 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a\": container with ID starting with c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a not found: ID does not exist" containerID="c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a" Jul 27 17:52:31.162192 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.162140 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a"} err="failed to get container status \"c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a\": rpc error: code = NotFound desc = could not find container \"c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a\": container with ID starting with c71bb003cb781644b50ba4e9364fd453474e0dd908ca11f5472c1fcc248c2d2a not found: ID does not exist" Jul 27 17:52:31.167798 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.167775 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn"] Jul 27 17:52:31.171903 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:31.171876 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-004618-openshift-default-6c499d8568-25gcn"] Jul 27 17:52:32.064561 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:32.064525 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb83b28-897a-4917-8189-8740d286cf42" path="/var/lib/kubelet/pods/bcb83b28-897a-4917-8189-8740d286cf42/volumes" Jul 27 17:52:36.145217 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.145163 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:36.196894 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.196854 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn"] Jul 27 17:52:36.197406 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.197370 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" podUID="cb2aa048-5500-4cd1-9086-1f9a5e3e71af" containerName="payload-pre-processing" containerID="cri-o://72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae" gracePeriod=30 Jul 27 17:52:36.230842 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.230809 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:36.300288 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.300251 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:36.484099 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.484077 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:52:36.591452 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.591416 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-plugins-config-volume\") pod \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\" (UID: \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\") " Jul 27 17:52:36.591648 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.591525 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlvs2\" (UniqueName: \"kubernetes.io/projected/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-kube-api-access-mlvs2\") pod \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\" (UID: \"cb2aa048-5500-4cd1-9086-1f9a5e3e71af\") " Jul 27 17:52:36.591812 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.591787 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb2aa048-5500-4cd1-9086-1f9a5e3e71af" (UID: "cb2aa048-5500-4cd1-9086-1f9a5e3e71af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:52:36.593654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.593628 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-kube-api-access-mlvs2" (OuterVolumeSpecName: "kube-api-access-mlvs2") pod "cb2aa048-5500-4cd1-9086-1f9a5e3e71af" (UID: "cb2aa048-5500-4cd1-9086-1f9a5e3e71af"). InnerVolumeSpecName "kube-api-access-mlvs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:52:36.692814 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.692771 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-plugins-config-volume\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:36.692814 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:36.692803 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvs2\" (UniqueName: \"kubernetes.io/projected/cb2aa048-5500-4cd1-9086-1f9a5e3e71af-kube-api-access-mlvs2\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:37.175492 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.175459 2557 generic.go:356] "Generic (PLEG): container finished" podID="cb2aa048-5500-4cd1-9086-1f9a5e3e71af" containerID="72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae" exitCode=0 Jul 27 17:52:37.175947 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.175498 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" event={"ID":"cb2aa048-5500-4cd1-9086-1f9a5e3e71af","Type":"ContainerDied","Data":"72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae"} Jul 27 17:52:37.175947 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.175559 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" event={"ID":"cb2aa048-5500-4cd1-9086-1f9a5e3e71af","Type":"ContainerDied","Data":"37c1ca12deb8b18fe6ad8b120a1c0ff95c7814c17263662cb7b747162366fdb8"} Jul 27 17:52:37.175947 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.175510 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn" Jul 27 17:52:37.175947 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.175582 2557 scope.go:117] "RemoveContainer" containerID="72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae" Jul 27 17:52:37.187211 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.187167 2557 scope.go:117] "RemoveContainer" containerID="72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae" Jul 27 17:52:37.187719 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:52:37.187696 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae\": container with ID starting with 72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae not found: ID does not exist" containerID="72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae" Jul 27 17:52:37.187799 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.187730 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae"} err="failed to get container status \"72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae\": rpc error: code = NotFound desc = could not find container \"72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae\": container with ID starting with 72ea3b42b961c6a26e13b16d673c3f256b3bef2a38f18277283d7c532a0a98ae not found: ID does not exist" Jul 27 17:52:37.201334 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.201282 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn"] Jul 27 17:52:37.203739 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:37.203716 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-509a5d-5f98868d7b-b54rn"] Jul 27 17:52:38.064869 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:38.064833 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2aa048-5500-4cd1-9086-1f9a5e3e71af" path="/var/lib/kubelet/pods/cb2aa048-5500-4cd1-9086-1f9a5e3e71af/volumes" Jul 27 17:52:40.628907 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:40.628873 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:41.009484 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:41.009367 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:47.441851 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:47.441813 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:47.692921 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:47.692835 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7"] Jul 27 17:52:47.693151 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:47.693089 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" containerID="cri-o://d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5" gracePeriod=30 Jul 27 17:52:49.024353 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:49.024312 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:52.749398 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:52:52.749362 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04338905_db2a_4e0c_b599_ca4e2c5ba11c.slice/crio-conmon-d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5.scope\": RecentStats: unable to find data in memory cache]" Jul 27 17:52:52.957199 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:52.957116 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:52:53.036651 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036616 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrp26\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-kube-api-access-mrp26\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.036834 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036667 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-envoy\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.036834 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036704 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-data\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.036834 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036723 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-socket\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.036834 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036743 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-certs\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.036834 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036771 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istiod-ca-cert\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.036834 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036807 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-credential-socket\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.037147 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036852 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-token\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.037147 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.036895 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-podinfo\") pod \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\" (UID: \"04338905-db2a-4e0c-b599-ca4e2c5ba11c\") " Jul 27 17:52:53.037302 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.037145 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:53.037359 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.037315 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:52:53.037469 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.037435 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:53.037611 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.037584 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-data" (OuterVolumeSpecName: "istio-data") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:53.037695 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.037593 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:53.039225 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.039199 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:52:53.039225 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.039213 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 17:52:53.039463 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.039444 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-kube-api-access-mrp26" (OuterVolumeSpecName: "kube-api-access-mrp26") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "kube-api-access-mrp26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:52:53.039517 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.039471 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-token" (OuterVolumeSpecName: "istio-token") pod "04338905-db2a-4e0c-b599-ca4e2c5ba11c" (UID: "04338905-db2a-4e0c-b599-ca4e2c5ba11c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:52:53.138228 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138164 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-token\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.138228 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138222 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-podinfo\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.138228 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138232 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrp26\" (UniqueName: \"kubernetes.io/projected/04338905-db2a-4e0c-b599-ca4e2c5ba11c-kube-api-access-mrp26\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.138471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138243 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-envoy\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.138471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138252 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istio-data\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.138471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138260 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.138471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138267 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-workload-certs\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.138471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138275 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04338905-db2a-4e0c-b599-ca4e2c5ba11c-istiod-ca-cert\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.138471 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.138282 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04338905-db2a-4e0c-b599-ca4e2c5ba11c-credential-socket\") on node \"ip-10-0-128-46.ec2.internal\" DevicePath \"\"" Jul 27 17:52:53.249473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.249387 2557 generic.go:356] "Generic (PLEG): container finished" podID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerID="d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5" exitCode=0 Jul 27 17:52:53.249473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.249438 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" Jul 27 17:52:53.249657 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.249471 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" event={"ID":"04338905-db2a-4e0c-b599-ca4e2c5ba11c","Type":"ContainerDied","Data":"d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5"} Jul 27 17:52:53.249657 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.249526 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7" event={"ID":"04338905-db2a-4e0c-b599-ca4e2c5ba11c","Type":"ContainerDied","Data":"6dfdd87ae9a552bde95092722d93d4d2105f9eb2318432c474796c20700891bb"} Jul 27 17:52:53.249657 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.249550 2557 scope.go:117] "RemoveContainer" containerID="d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5" Jul 27 17:52:53.258922 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.258906 2557 scope.go:117] "RemoveContainer" containerID="bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b" Jul 27 17:52:53.266897 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.266880 2557 scope.go:117] "RemoveContainer" containerID="d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5" Jul 27 17:52:53.267145 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:52:53.267123 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5\": container with ID starting with d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5 not found: ID does not exist" containerID="d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5" Jul 27 17:52:53.267228 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.267153 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5"} err="failed to get container status \"d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5\": rpc error: code = NotFound desc = could not find container \"d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5\": container with ID starting with d8b86233defe2455329dc47126a494f85cf2ed7a00f9356728ecd21d10d8e5c5 not found: ID does not exist" Jul 27 17:52:53.267228 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.267170 2557 scope.go:117] "RemoveContainer" containerID="bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b" Jul 27 17:52:53.267417 ip-10-0-128-46 kubenswrapper[2557]: E0727 17:52:53.267399 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b\": container with ID starting with bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b not found: ID does not exist" containerID="bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b" Jul 27 17:52:53.267458 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.267423 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b"} err="failed to get container status \"bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b\": rpc error: code = NotFound desc = could not find container \"bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b\": container with ID starting with bd67edf2b3dc08b2194cc1c6fe5b9ce37014fd7db3b0acf65710a119fdf00b7b not found: ID does not exist" Jul 27 17:52:53.273183 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.273146 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7"] Jul 27 17:52:53.276195 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.276163 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-509a5d-openshift-default-6d968bc6dd-r74x7"] Jul 27 17:52:53.943804 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:53.943777 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d2wq9_39fd0a82-20a5-4271-90f7-a3fcbccc0d29/manager/0.log" Jul 27 17:52:54.066965 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:54.066899 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" path="/var/lib/kubelet/pods/04338905-db2a-4e0c-b599-ca4e2c5ba11c/volumes" Jul 27 17:52:54.069217 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:54.069190 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-s5z5z_da8ded80-b230-44e4-97c6-59f49d5e4c56/manager/0.log" Jul 27 17:52:54.192064 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:54.192035 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74cbc56ff7-2kjbn_f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd/manager/0.log" Jul 27 17:52:54.302363 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:54.302287 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-z8zwd_879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5/server/0.log" Jul 27 17:52:54.419646 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:54.419613 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pj8fh_2109efaa-eafa-4c8b-a66c-98c97c873333/manager/0.log" Jul 27 17:52:54.781030 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:54.780990 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c484996-lshg7_969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1/manager/0.log" Jul 27 17:52:54.792544 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:54.792518 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c484996-lshg7_969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1/copy-manifests/0.log" Jul 27 17:52:55.271449 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:55.271421 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29752875-lr4bn_563d6016-d5ea-4b7f-8fb4-6b048c05047b/cleanup/0.log" Jul 27 17:52:55.382165 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:55.382139 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29752890-557qb_aef23af4-0e30-4bea-850c-3b25673d1e5c/cleanup/0.log" Jul 27 17:52:55.609230 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:55.609131 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-pcpqk_5a1f7668-1bcb-4a9f-ad3a-b87c0d8bcc05/postgres/0.log" Jul 27 17:52:58.162239 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:58.162195 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wwxbz_4ec4f2f3-2370-4089-9299-993253494855/discovery/0.log" Jul 27 17:52:58.295646 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:58.295608 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc796b767-hvjkh_bef354c2-610b-453c-a659-5075fb9afeb7/kube-auth-proxy/0.log" Jul 27 17:52:58.524768 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:58.524690 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d7q5z_8264e4fa-d20a-4d04-9fd2-384a1cc596b6/istio-proxy/1.log" Jul 27 17:52:58.643107 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:58.643077 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-bjtcc_7105d860-e49f-470e-a9ee-a0773eee1584/payload-pre-processing/0.log" Jul 27 17:52:58.770942 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:58.770912 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vcgrx_69dfa449-4401-4cf9-8374-0f08e94eeeae/payload-processing/0.log" Jul 27 17:52:59.397385 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:59.397351 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vfsgm_3b75df30-f5b2-4906-b499-42a88f0e1346/main/0.log" Jul 27 17:52:59.502186 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:59.502146 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-trzsd_2639b0f8-517e-4cfb-b3bb-6569c963ef17/main/0.log" Jul 27 17:52:59.612400 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:52:59.612370 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgnp2_123ae25c-8068-40c2-b9eb-9c5e295bcc9d/main/0.log" Jul 27 17:53:05.893534 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:05.893486 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:53:05.954619 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:05.954586 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" limit=1024 Jul 27 17:53:08.739058 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:08.739019 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t54n4_2402f312-cad4-44fa-a9ae-b752b73c6eb3/global-pull-secret-syncer/0.log" Jul 27 17:53:08.805587 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:08.805559 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jm5gm_675a1b63-8414-428b-a1ec-4084d018e04c/konnectivity-agent/0.log" Jul 27 17:53:08.908107 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:08.908070 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-46.ec2.internal_e07fcea3b53729941e121c6d1709b9c1/haproxy/0.log" Jul 27 17:53:17.543897 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:17.543869 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2tl8_315c980f-4d4d-4c47-b2b0-00a20854279e/node-exporter/0.log" Jul 27 17:53:17.580543 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:17.580519 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2tl8_315c980f-4d4d-4c47-b2b0-00a20854279e/kube-rbac-proxy/0.log" Jul 27 17:53:17.606680 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:17.606658 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2tl8_315c980f-4d4d-4c47-b2b0-00a20854279e/init-textfile/0.log" Jul 27 17:53:18.371067 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371033 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7"] Jul 27 17:53:18.371473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371446 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" Jul 27 17:53:18.371473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371464 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" Jul 27 17:53:18.371473 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371476 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371482 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371490 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371495 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371506 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371515 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371525 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371530 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371540 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce55b173-3f4e-4782-a714-eece9ad9d566" containerName="payload-pre-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371546 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce55b173-3f4e-4782-a714-eece9ad9d566" containerName="payload-pre-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371553 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2aa048-5500-4cd1-9086-1f9a5e3e71af" containerName="payload-pre-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371557 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2aa048-5500-4cd1-9086-1f9a5e3e71af" containerName="payload-pre-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371563 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10ad513-25ca-4881-be86-b659e4ce74d1" containerName="payload-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371568 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10ad513-25ca-4881-be86-b659e4ce74d1" containerName="payload-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371574 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3623ccbb-f221-4f54-b4a0-f0363b0cb97a" containerName="payload-pre-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371580 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3623ccbb-f221-4f54-b4a0-f0363b0cb97a" containerName="payload-pre-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371634 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371641 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf380c91-83f8-44e8-be87-50eff59bfc6c" containerName="istio-proxy" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371647 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce55b173-3f4e-4782-a714-eece9ad9d566" containerName="payload-pre-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371654 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10ad513-25ca-4881-be86-b659e4ce74d1" containerName="payload-processing" Jul 27 17:53:18.371654 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371660 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2aa048-5500-4cd1-9086-1f9a5e3e71af" containerName="payload-pre-processing" Jul 27 17:53:18.372280 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371672 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3623ccbb-f221-4f54-b4a0-f0363b0cb97a" containerName="payload-pre-processing" Jul 27 17:53:18.372280 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371681 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="04338905-db2a-4e0c-b599-ca4e2c5ba11c" containerName="istio-proxy" Jul 27 17:53:18.372280 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.371686 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb83b28-897a-4917-8189-8740d286cf42" containerName="istio-proxy" Jul 27 17:53:18.374630 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.374614 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.380170 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.380148 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tn87g\"/\"default-dockercfg-78w79\"" Jul 27 17:53:18.381076 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.380833 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tn87g\"/\"kube-root-ca.crt\"" Jul 27 17:53:18.381076 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.381065 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tn87g\"/\"openshift-service-ca.crt\"" Jul 27 17:53:18.387984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.387959 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7"] Jul 27 17:53:18.450595 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.450559 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-lib-modules\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.450595 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.450593 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-sys\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.450866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.450631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-podres\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.450866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.450719 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-proc\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.450866 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.450785 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhgc\" (UniqueName: \"kubernetes.io/projected/b8acf5fb-7aa5-4004-84df-3529ea32a110-kube-api-access-kfhgc\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551370 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551329 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhgc\" (UniqueName: \"kubernetes.io/projected/b8acf5fb-7aa5-4004-84df-3529ea32a110-kube-api-access-kfhgc\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551389 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-lib-modules\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-sys\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551458 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-podres\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551524 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-proc\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551543 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-sys\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551571 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-lib-modules\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551634 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-proc\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.551827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.551651 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8acf5fb-7aa5-4004-84df-3529ea32a110-podres\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.562128 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.562105 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhgc\" (UniqueName: \"kubernetes.io/projected/b8acf5fb-7aa5-4004-84df-3529ea32a110-kube-api-access-kfhgc\") pod \"perf-node-gather-daemonset-qmzt7\" (UID: \"b8acf5fb-7aa5-4004-84df-3529ea32a110\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.684339 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.684304 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:18.828604 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:18.828577 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7"] Jul 27 17:53:18.830428 ip-10-0-128-46 kubenswrapper[2557]: W0727 17:53:18.830384 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb8acf5fb_7aa5_4004_84df_3529ea32a110.slice/crio-1fc7274e7f0005ce6d8ccc0677a1de71ea5bc60b214d744c5e9afcfe9db96330 WatchSource:0}: Error finding container 1fc7274e7f0005ce6d8ccc0677a1de71ea5bc60b214d744c5e9afcfe9db96330: Status 404 returned error can't find the container with id 1fc7274e7f0005ce6d8ccc0677a1de71ea5bc60b214d744c5e9afcfe9db96330 Jul 27 17:53:19.368864 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:19.368762 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" event={"ID":"b8acf5fb-7aa5-4004-84df-3529ea32a110","Type":"ContainerStarted","Data":"760c941de2d95d350a773413f727ae4c2a1c3b67ffe370e84fdd51a5d5be71eb"} Jul 27 17:53:19.368864 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:19.368827 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" event={"ID":"b8acf5fb-7aa5-4004-84df-3529ea32a110","Type":"ContainerStarted","Data":"1fc7274e7f0005ce6d8ccc0677a1de71ea5bc60b214d744c5e9afcfe9db96330"} Jul 27 17:53:19.369070 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:19.368942 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:23.531643 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:23.531612 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sdkgm_61ac0229-6cf5-4b4d-a406-42101ea16da7/dns/0.log" Jul 27 17:53:23.555407 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:23.555380 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sdkgm_61ac0229-6cf5-4b4d-a406-42101ea16da7/kube-rbac-proxy/0.log" Jul 27 17:53:23.670303 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:23.670270 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wv5xc_87e0c5f4-8b4b-4164-9d12-ebafec319642/dns-node-resolver/0.log" Jul 27 17:53:24.423192 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:24.423145 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7gcp5_9209f648-9b67-4793-aeb6-250869f631b9/node-ca/0.log" Jul 27 17:53:25.382664 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:25.382638 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" Jul 27 17:53:25.400646 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:25.400599 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-qmzt7" podStartSLOduration=7.4005784949999995 podStartE2EDuration="7.400578495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:53:19.402764605 +0000 UTC m=+3395.873090813" watchObservedRunningTime="2026-07-27 17:53:25.400578495 +0000 UTC m=+3401.870904689" Jul 27 17:53:25.771094 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:25.771008 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wwxbz_4ec4f2f3-2370-4089-9299-993253494855/discovery/0.log" Jul 27 17:53:25.804264 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:25.804233 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc796b767-hvjkh_bef354c2-610b-453c-a659-5075fb9afeb7/kube-auth-proxy/0.log" Jul 27 17:53:25.887968 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:25.887938 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d7q5z_8264e4fa-d20a-4d04-9fd2-384a1cc596b6/istio-proxy/0.log" Jul 27 17:53:25.937435 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:25.937404 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d7q5z_8264e4fa-d20a-4d04-9fd2-384a1cc596b6/istio-proxy/1.log" Jul 27 17:53:26.016139 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:26.016104 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-bjtcc_7105d860-e49f-470e-a9ee-a0773eee1584/payload-pre-processing/0.log" Jul 27 17:53:26.062827 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:26.062741 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vcgrx_69dfa449-4401-4cf9-8374-0f08e94eeeae/payload-processing/0.log" Jul 27 17:53:27.047344 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:27.047306 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7hlm6_3b2789e2-beaf-4d1f-b5ff-8df0a8f53979/serve-healthcheck-canary/0.log" Jul 27 17:53:27.880831 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:27.880800 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lr9b8_c12e97b8-48e3-4341-a8b5-700608955f33/kube-rbac-proxy/0.log" Jul 27 17:53:27.911973 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:27.911947 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lr9b8_c12e97b8-48e3-4341-a8b5-700608955f33/exporter/0.log" Jul 27 17:53:27.950777 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:27.950753 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lr9b8_c12e97b8-48e3-4341-a8b5-700608955f33/extractor/0.log" Jul 27 17:53:28.591268 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:28.591168 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wwxbz_4ec4f2f3-2370-4089-9299-993253494855/discovery/0.log" Jul 27 17:53:28.611090 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:28.611059 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc796b767-hvjkh_bef354c2-610b-453c-a659-5075fb9afeb7/kube-auth-proxy/0.log" Jul 27 17:53:28.667896 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:28.667860 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d7q5z_8264e4fa-d20a-4d04-9fd2-384a1cc596b6/istio-proxy/0.log" Jul 27 17:53:28.716570 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:28.716542 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d7q5z_8264e4fa-d20a-4d04-9fd2-384a1cc596b6/istio-proxy/1.log" Jul 27 17:53:28.764584 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:28.764559 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-bjtcc_7105d860-e49f-470e-a9ee-a0773eee1584/payload-pre-processing/0.log" Jul 27 17:53:28.793840 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:28.793814 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vcgrx_69dfa449-4401-4cf9-8374-0f08e94eeeae/payload-processing/0.log" Jul 27 17:53:30.412984 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:30.412947 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wwxbz_4ec4f2f3-2370-4089-9299-993253494855/discovery/0.log" Jul 27 17:53:30.435343 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:30.435315 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc796b767-hvjkh_bef354c2-610b-453c-a659-5075fb9afeb7/kube-auth-proxy/0.log" Jul 27 17:53:30.449487 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:30.449465 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d2wq9_39fd0a82-20a5-4271-90f7-a3fcbccc0d29/manager/0.log" Jul 27 17:53:30.477786 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:30.477704 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d7q5z_8264e4fa-d20a-4d04-9fd2-384a1cc596b6/istio-proxy/0.log" Jul 27 17:53:30.551523 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:30.551494 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d7q5z_8264e4fa-d20a-4d04-9fd2-384a1cc596b6/istio-proxy/1.log" Jul 27 17:53:30.598613 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:30.598560 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-bjtcc_7105d860-e49f-470e-a9ee-a0773eee1584/payload-pre-processing/0.log" Jul 27 17:53:30.636242 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:30.636209 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vcgrx_69dfa449-4401-4cf9-8374-0f08e94eeeae/payload-processing/0.log" Jul 27 17:53:30.865446 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:30.865362 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-s5z5z_da8ded80-b230-44e4-97c6-59f49d5e4c56/manager/0.log" Jul 27 17:53:31.127058 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:31.127025 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74cbc56ff7-2kjbn_f1dfe1a1-6e69-4cc4-bf01-98bf4a4a9dbd/manager/0.log" Jul 27 17:53:31.162903 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:31.162871 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-z8zwd_879bf6a2-16bf-40e9-8cc2-87e4eef8b7e5/server/0.log" Jul 27 17:53:31.284991 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:31.284954 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pj8fh_2109efaa-eafa-4c8b-a66c-98c97c873333/manager/0.log" Jul 27 17:53:31.646000 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:31.645970 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c484996-lshg7_969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1/manager/0.log" Jul 27 17:53:31.693033 ip-10-0-128-46 kubenswrapper[2557]: I0727 17:53:31.693008 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c484996-lshg7_969a1945-f7c9-4ffa-bfa0-2c508d6bfaf1/copy-manifests/0.log"